{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "joined-glasgow",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# 类对象诞生记"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "institutional-oliver",
   "metadata": {
    "hidden": true
   },
   "source": [
    "从本节开始，我们着手研究 Python 面向对象中自定义类的运行机制，并借此实现一些有趣的应用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italic-eligibility",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 类对象行为"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "structural-pepper",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们从最简单的类入手，研究类对象和实例对象的创建过程以及典型行为。还记得我们在 对象模型 部分中，作为例子讲解的 Dog 类吗？这类麻雀虽小，却五脏俱全，非常适合作为我们研究的起点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "stylish-stuff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:21:41.990795Z",
     "start_time": "2021-03-29T12:21:41.974796Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "woof\n"
     ]
    }
   ],
   "source": [
    ">>> class Dog:\n",
    "...     def yelp(self):\n",
    "...         print('woof')\n",
    "...\n",
    ">>> Dog\n",
    ">>> dog = Dog()\n",
    ">>> dog.yelp()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "monetary-animation",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据 对象模型 部分学到的知识，我们可以得到这样的关系图：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_01.png\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "combined-breath",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:22:22.596505Z",
     "start_time": "2021-03-29T12:22:22.578508Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> type(dog) is Dog"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "double-moment",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:22:23.530422Z",
     "start_time": "2021-03-29T12:22:23.521421Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> type(Dog) is type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "valued-third",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:22:24.407697Z",
     "start_time": "2021-03-29T12:22:24.400699Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> issubclass(Dog, object)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "departmental-tender",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果你对 类型 对象、 实例 对象、 继承 、 实例化 等面向对象概念印象不深，请复习 对象模型 部分章节。\n",
    "\n",
    "我们接着观察 Dog 类的行为，发现它支持属性设置。例如，我们设置 legs 属性，表示狗都有 4 条腿："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "qualified-funeral",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:22:55.985525Z",
     "start_time": "2021-03-29T12:22:55.977530Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> Dog.legs = 4\n",
    ">>> Dog.legs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appreciated-protein",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由此，我们可以大胆推测，每个自定义类对象都有一个 属性空间 ，背后同样是由 dict 对象实现的。\n",
    "\n",
    "我们还注意到类对象中的属性，可以被实例对象访问到。在 继承与属性查找 一节，我们将深入研究这个现象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "lesser-pendant",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:23:49.077665Z",
     "start_time": "2021-03-29T12:23:49.057665Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> dog = Dog()\n",
    ">>> dog.legs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sacred-passion",
   "metadata": {
    "hidden": true
   },
   "source": [
    "那么，类方法是不是也保存在类属性空间中呢？我们知道，通过 __dict__ 即可获取一个对象的属性空间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "capital-relations",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:24:03.090412Z",
     "start_time": "2021-03-29T12:24:03.076413Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['__module__', 'yelp', '__dict__', '__weakref__', '__doc__', 'legs'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> Dog.__dict__.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "looking-minnesota",
   "metadata": {
    "hidden": true
   },
   "source": [
    "除了几个内置属性，我们找到了 legs 这是我们刚刚设置的属性；还找到了 yelp ，它应该就是类的方法了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "micro-california",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:24:29.985637Z",
     "start_time": "2021-03-29T12:24:29.977636Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.Dog.yelp(self)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> Dog.__dict__['yelp']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "threatened-bacteria",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:24:54.184609Z",
     "start_time": "2021-03-29T12:24:54.169609Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.Dog.yelp(self)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Dog.yelp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaged-builder",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这是否意味着，可以将新方法作为属性设置到类对象身上，让它具有某些新行为呢？事不宜迟，我们来试试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "national-hollow",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:25:26.269646Z",
     "start_time": "2021-03-29T12:25:26.266645Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> def yelp2(self):\n",
    "...     print('surprise')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "classified-scale",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-29T12:25:30.256687Z",
     "start_time": "2021-03-29T12:25:30.241687Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "surprise\n"
     ]
    }
   ],
   "source": [
    ">>> Dog.yelp2 = yelp2\n",
    ">>> dog.yelp2()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "powered-huntington",
   "metadata": {
    "hidden": true
   },
   "source": [
    "哇，我们在运行时让 Dog 类拥有 yelp2 这个新能力，Python 果然很动态！这个例子帮我们理解 Python 的运行行为，但这种写法不是一个好习惯，在实际项目中尽量少用吧。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_02.png\" width=800 height=600>  \n",
    "\n",
    "至此，我们得到了 Dog 这个类对象的大致轮廓，它有一个属性空间，由 dict 对象实现， yelp 方法便位于其中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "filled-bouquet",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 底层表现形式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "developing-marketing",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Dog 类是一种自定义类对象，因此它底层应该是一个 PyTypeObject 。PyTypeObject 是每个类型对象在虚拟机底层的表现形式，它中源码中位于 Objects/typeobject.c 文件。这个结构体前面章节已有所涉猎，多少还有些印象吧？\n",
    "\n",
    "\n",
    "进入 Objects 目录，我们的目光还被 Objects/classobject.c 这个文件吸引住了，这不就是 类对象 吗？历史上，Python 自定义类和内建类型是分开实现的，也因此造成鸿沟。后来 Python 完成了类型统一，不管自定义的还是内建的，均由 PyTypeObject 实现，Objects/classobject.c 中的大部分功能也因之废弃。\n",
    "\n",
    "\n",
    "由于篇幅的关系，这里不打算展开太多源码细节，同样以最通俗易懂的结构图进行介绍\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_03.jpg\" width=800 height=600>  \n",
    "\n",
    "结构图省略了很多字段，但不会影响学习理解，重点注意这几个字段：\n",
    "\n",
    "- tp_name ，Dog 类对象名字，即类名；\n",
    "- tp_call ，Dog 类被调用时执行的函数指针，用于创建 Dog 实例对象 ，如 dog = Dog() ；\n",
    "- tp_dict ，该字段指向一个 dict 对象，dict 存储 Dog 类对象的属性空间；\n",
    "\n",
    "对源码比较感兴趣的同学，可以深入 Objects/typeobject.c 以及 Objects/classobject.c ，研究体会。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "congressional-hobby",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 创建步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "natural-burton",
   "metadata": {
    "hidden": true
   },
   "source": [
    "那么，Python 类代码时如何一步步转换成类对象的呢？同样，我们从字节码入手，一起窥探这其中的秘密。\n",
    "\n",
    "首先，我们将 Dog 类的代码作为字符串保存起来，并调用 compile 函数进行编译，得到一个代码对象：\n",
    "\n",
    "    >>> text = '''\n",
    "    ... class Dog:\n",
    "    ...     def yelp(self):\n",
    "    ...         print('woof')\n",
    "    ... '''\n",
    "    >>> code = compile(text, '', 'exec')\n",
    "    \n",
    "这个代码对象里头应该就保持着创建类对象的密码！迫不及待想要揭晓答案，那就字节码反编译一下吧：\n",
    "\n",
    "    >>> import dis\n",
    "    >>> dis.dis(code)\n",
    "      2           0 LOAD_BUILD_CLASS\n",
    "                  2 LOAD_CONST               0 (<code object Dog at 0x10110f8a0, file \"\", line 2>)\n",
    "                  4 LOAD_CONST               1 ('Dog')\n",
    "                  6 MAKE_FUNCTION            0\n",
    "                  8 LOAD_CONST               1 ('Dog')\n",
    "                 10 CALL_FUNCTION            2\n",
    "                 12 STORE_NAME               0 (Dog)\n",
    "                 14 LOAD_CONST               2 (None)\n",
    "                 16 RETURN_VALUE\n",
    "\n",
    "    Disassembly of <code object Dog at 0x10110f8a0, file \"\", line 2>:\n",
    "      2           0 LOAD_NAME                0 (__name__)\n",
    "                  2 STORE_NAME               1 (__module__)\n",
    "                  4 LOAD_CONST               0 ('Dog')\n",
    "                  6 STORE_NAME               2 (__qualname__)\n",
    "\n",
    "      3           8 LOAD_CONST               1 (<code object yelp at 0x10110f390, file \"\", line 3>)\n",
    "                 10 LOAD_CONST               2 ('Dog.yelp')\n",
    "                 12 MAKE_FUNCTION            0\n",
    "                 14 STORE_NAME               3 (yelp)\n",
    "                 16 LOAD_CONST               3 (None)\n",
    "                 18 RETURN_VALUE\n",
    "\n",
    "    Disassembly of <code object yelp at 0x10110f390, file \"\", line 3>:\n",
    "      4           0 LOAD_GLOBAL              0 (print)\n",
    "                  2 LOAD_CONST               1 ('woof')\n",
    "                  4 CALL_FUNCTION            1\n",
    "                  6 POP_TOP\n",
    "                  8 LOAD_CONST               0 (None)\n",
    "                 10 RETURN_VALUE\n",
    "                 \n",
    " 里头大部分字节码我们都认识，但是连起来看却一脸懵逼……这不打紧，至少我们已经发现了某些蛛丝马迹。看到字节码 LOAD_BUILD_CLASS 没？顾名思义，它应该就是构建类对象的关键所在。\n",
    "\n",
    "开始窥探 LOAD_BUILD_CLASS 字节码之前，先继续考察代表 Dog 类的代码对象。由于我们将 exec 参数传递给 compile 函数，将代码代码作为模块进行编译，因此代码对象 code 对应着模块级别代码块。\n",
    "\n",
    "我们发现，code 里面还藏有子代码对象，子代码对象作为常量存在于 code 常量表中。而子代码对象中藏在另一个代码对象，同样以常量的形式存在：\n",
    "\n",
    "    >>> code.co_consts\n",
    "    (<code object Dog at 0x10110f8a0, file \"\", line 2>, 'Dog', None)\n",
    "    >>> code.co_consts[0].co_consts\n",
    "    ('Dog', <code object yelp at 0x10110f390, file \"\", line 3>, 'Dog.yelp', None)\n",
    "    \n",
    "从这三个代码对象的字节码来看，code 对应着模块代码，也就是最外层代码块；code.co_consts[0] 则对应着 Dog 类代码块；而 code.co_consts[0].co_consts[1] 则对应着 yelp 函数代码块。三者关系如下图：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_04.jpg\" width=800 height=600>  \n",
    "\n",
    "代码对象作为源码编译的结果，与源码在层级上一一对应，堪称完美。那么，这些毫无生命力的代码对象，又是如何一步步变身为活生生的类对象的呢？我们得深入字节码中寻找答案。\n",
    "\n",
    "接下来，我们一起来推演，虚拟机逐条执行模块字节码之后，发生了什么神奇的事情！当模块代码执行时，虚拟机内部有一个 **栈帧** 对象，维护着代码对象运行时时的上下文信息，全局和局部名字空间均指向模块属性空间。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_05.jpg\" width=800 height=600>  \n",
    "\n",
    "注意到，前 3 条字节码，负责往运行栈加载数据。LOAD_BUILD_CLASS 这个字节码很特殊，我们第一次遇到。从 Python/ceval.c 源码，我们发现他的作用很简单，只是将 __build_class__ 这个内建函数加载进栈顶。该函数为 builtins 模块中的一员，它的名字告诉我们，这是一个负责创建类的工具函数。\n",
    "\n",
    "接着，第 2 条字节码将下标为 0 的常量加载到栈顶，这是代表 Dog 类代码块的代码对象，如红色箭头所示。第 3 条字节码将常量 Dog 加载到栈顶。第 4 条字码时我们在函数机制中学过的 MAKE_FUNCTION 指令，用于创建函数对象。指令执行完毕后，名为 Dog 的函数对象就被保存在栈顶了，如图粉红色部分：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_06.jpg\" width=800 height=600>  \n",
    "\n",
    "这就很诡异，为啥这里需要创建一个函数对象呢？我们接着推演。第 5 条字节码将常量 Dog 加载到栈顶；第 6 条字节码调用 __build_class__ 函数。请特别注意，这里调用的不是刚刚创建的 Dog 函数，它只是作为参数传给了 __build_class__ 函数。从 __build_class__ 函数的帮助文档得知，该函数第一个参数为一个函数，第二个参数为一个名字，跟我们的分析是吻合的。\n",
    "\n",
    "    >>> help(__build_class__)\n",
    "    Help on built-in function __build_class__ in module builtins:\n",
    "\n",
    "    __build_class__(...)\n",
    "        __build_class__(func, name, *bases, metaclass=None, **kwds) -> class\n",
    "\n",
    "        Internal helper function used by the class statement.\n",
    "        \n",
    "那么，Dog 函数是在 __build_class__ 中被调用的吗？它的作用又是什么呢？我们接着扒开 __build_class__ 的源码看一看，它位于 builtins 模块中，源码路径为 Python/bltinmodule.c 。\n",
    "\n",
    "我们发现，__build_class__ 并没有直接调用新创建的 Dog 函数。它先创建一个 dict 对象，作为新生类的属性空间；然后，从 Dog 函数中取出全局名字空间和代码对象；最后新建一个栈帧对象并开始执行 Dog 类代码对象。\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_07.jpg\" width=800 height=600>  \n",
    "\n",
    "注意到，新栈帧对象从从 Dog 函数对象中取得全局名字空间，这也是模块的属性空间；又从函数对象取得代码对象，这个代码对象正好对应着 Dog 类的代码块；而局部名字空间则是新生类的属性空间。因此，Dog 这个函数对象只是作为打包参数的包袱，将代码对象、全局名字空间等参数作为一个整体进行传递，多么巧妙！\n",
    "\n",
    "Dog 类代码对象的字节码我们已经非常熟悉了，接着推演一番。代码先从全局名字空间取出模块名 __name__ ，在局部名字空间中保存为 __module__ ，原来类对象的 __module__ 属性就是这么来的！然后将类名 Dog 保存到局部名字空间中的 __qualname__ 。最后取出 yelp 函数的代码对象，完成 yelp 函数对象的创建工作。至此，新生类 Dog 的属性空间完成初始化：\n",
    "\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_08.jpg\" width=800 height=600>  \n",
    "\n",
    "类属性空间准备完毕，__build_class__ 接着调用 type 元类型对象完成 Dog 类的创建。type 需要的参数有 3个，分别是： 类名 、 基类列表 (类继承)以及代表 类属性空间 的 dict 对象。\n",
    "\n",
    "    >>> help(type)\n",
    "    Help on class type in module builtins:\n",
    "\n",
    "    class type(object)\n",
    "     |  type(object_or_name, bases, dict)\n",
    "     |  type(object) -> the object's type\n",
    "     |  type(name, bases, dict) -> a new type\n",
    "     \n",
    " 至此，Dog 类对象横空出世！\n",
    " \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_09.jpg\" width=800 height=600>  \n",
    " \n",
    " 模块代码对象最后的字节码将 Dog 类对象保存于模块属性空间，我们已经很熟悉就不再赘述了：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_10.jpg\" width=800 height=600>  \n",
    " \n",
    " 最后，我们以 Python 的语言来回顾类对象创建过程中的关键步骤，以此加深理解。\n",
    " \n",
    " 还记得吗？模块代码对象中包含类代码对象，而类代码对象中又包含着类函数的代码对象：\n",
    "\n",
    "    # 模块代码对象\n",
    "    >>> code\n",
    "    <code object <module> at 0x10b42e8a0, file \"\", line 2>\n",
    "    # Dog 类代码对象\n",
    "    >>> code.co_consts[0]\n",
    "    <code object Dog at 0x10b42ac00, file \"\", line 2>\n",
    "    # Dog 类 yelp 函数代码对象\n",
    "    >>> code.co_consts[0].co_consts[1]\n",
    "    <code object yelp at 0x10b5136f0, file \"\", line 3>\n",
    "    \n",
    "接着，新建一个 dict 对象，作为类的属性空间：\n",
    "\n",
    "    >>> attrs = {}\n",
    "    \n",
    "以类属性空间为局部名字空间，执行类代码对象，以此完成新类属性空间初始化：\n",
    "\n",
    "    >>> exec(code.co_consts[0], globals(), attrs)\n",
    "    >>> attrs\n",
    "    {'__module__': '__main__', '__qualname__': 'Dog', 'yelp': <function Dog.yelp at 0x10b732e18>}\n",
    "    \n",
    "最后，调用 type 函数完成类对象创建，由于 Dog 没有显式继承关系，基类列表为空：\n",
    "\n",
    "    >>> Dog = type('Dog', (), attrs)\n",
    "    >>> Dog\n",
    "    <class '__main__.Dog'>\n",
    "    \n",
    "哇！我们以一种全新的方式得到一个全新的类！\n",
    "\n",
    "    >>> dog = Dog()\n",
    "    >>> dog.yelp()\n",
    "    woof\n",
    "    \n",
    "掌握这些原理之后，我们后续可以干很多不可思议的事情呢！敬请期待。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "particular-clothing",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 类继承机制与属性查找"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recent-estimate",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 单继承"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appreciated-swaziland",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们在对象模型中，通过 Dog 与 Sleuth 类来讲解类继承关系。为了进一步讨论类继承机制，我们进一步扩充：\n",
    "\n",
    "    class Animal:\n",
    "        def run(self):\n",
    "            print('running')\n",
    "\n",
    "    class Dog(Animal):\n",
    "        def yelp(self):\n",
    "            print('woof')\n",
    "        def play(self):\n",
    "            print('playing')\n",
    "\n",
    "    class Sleuth(Dog):\n",
    "        def yelp(self):\n",
    "            print('WOOF!')\n",
    "        def hunt(self):\n",
    "            print('hunting')\n",
    "            \n",
    "通过引入 Animal 类，我们得到一条包含 3 个类的继承链，继承链结束语 object 基类型对象：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_11.jpg\" width=800 height=600>  \n",
    " \n",
    " 现在，实例化一个 Sleuth 对象，它可以调用自己定义的方法，例如 hunt ：\n",
    "\n",
    "    >>> s = Sleuth()\n",
    "    >>> s.hunt()\n",
    "    hunting\n",
    "    \n",
    "当然了，由于 Sleuth 类继承于 Dog 类，因此 Sleuth 对象也可以调用 Dog 定义的方法，例如 play ：\n",
    "\n",
    "    >>> s.play()\n",
    "    playing\n",
    "    \n",
    "Sleuth 类通过 Dog 类间接继承于 Animal 类，因此它也可以调用 Animal 定义的方法：\n",
    "\n",
    "    >>> s.run()\n",
    "    running\n",
    "    \n",
    "如果子类对父类中的方法不满意，还可以进行方法重写。猎犬吠声与普通狗有所不同，我们可以为 Sleuth 类重写 yelp 方法，以大写突出吠声的威武雄壮。这样一来，Sleuth 实例对象将执行 Sleuth 类中定义的 yelp 方法版本：\n",
    "\n",
    "    >>> s.yelp()\n",
    "    WOOF!\n",
    "    \n",
    "那么，Python 虚拟机内部是如何实现继承机制的呢？我们接着到字节码中寻找秘密。\n",
    "\n",
    "对以上例子进行编译，我们可以得到这样的字节码： \n",
    "\n",
    "      1           0 LOAD_BUILD_CLASS\n",
    "                  2 LOAD_CONST               0 (<code object Animal at 0x109b90810, file \"\", line 1>)\n",
    "                  4 LOAD_CONST               1 ('Animal')\n",
    "                  6 MAKE_FUNCTION            0\n",
    "                  8 LOAD_CONST               1 ('Animal')\n",
    "                 10 CALL_FUNCTION            2\n",
    "                 12 STORE_NAME               0 (Animal)\n",
    "\n",
    "      5          14 LOAD_BUILD_CLASS\n",
    "                 16 LOAD_CONST               2 (<code object Dog at 0x109bd1c90, file \"\", line 5>)\n",
    "                 18 LOAD_CONST               3 ('Dog')\n",
    "                 20 MAKE_FUNCTION            0\n",
    "                 22 LOAD_CONST               3 ('Dog')\n",
    "                 24 LOAD_NAME                0 (Animal)\n",
    "                 26 CALL_FUNCTION            3\n",
    "                 28 STORE_NAME               1 (Dog)\n",
    "\n",
    "     11          30 LOAD_BUILD_CLASS\n",
    "                 32 LOAD_CONST               4 (<code object Sleuth at 0x109bd1a50, file \"\", line 11>)\n",
    "                 34 LOAD_CONST               5 ('Sleuth')\n",
    "                 36 MAKE_FUNCTION            0\n",
    "                 38 LOAD_CONST               5 ('Sleuth')\n",
    "                 40 LOAD_NAME                1 (Dog)\n",
    "                 42 CALL_FUNCTION            3\n",
    "                 44 STORE_NAME               2 (Sleuth)\n",
    "                 46 LOAD_CONST               6 (None)\n",
    "                 48 RETURN_VALUE\n",
    "                                  \n",
    " 由上一小节，我们知道 LOAD_BUILD_CLASS 字节码用于加载 __build_class__ 函数，它创建类对象，接口如下：\n",
    "\n",
    "    >>> help(__build_class__)\n",
    "    Help on built-in function __build_class__ in module builtins:\n",
    "\n",
    "    __build_class__(...)\n",
    "        __build_class__(func, name, *bases, metaclass=None, **kwds) -> class\n",
    "\n",
    "        Internal helper function used by the class statement.\n",
    "        \n",
    "__build_class__ 相关参数如下：\n",
    "\n",
    "- func ，用于初始化类属性空间的可调动对象，由类代码块生成；\n",
    "- name ，类名；\n",
    "- bases ，基类，可以为多个；\n",
    "\n",
    "由此可见，创建子类时，需要将父类作为bases参数传给 __build_class__ 函数。\n",
    "\n",
    "创建 Animal 类时，由于没有显式指定继承关系，因此没有给 __build_class__ 函数传递任何基类：\n",
    "\n",
    "      1           0 LOAD_BUILD_CLASS\n",
    "                  2 LOAD_CONST               0 (<code object Animal at 0x109b90810, file \"\", line 1>)\n",
    "                  4 LOAD_CONST               1 ('Animal')\n",
    "                  6 MAKE_FUNCTION            0\n",
    "                  8 LOAD_CONST               1 ('Animal')\n",
    "                 10 CALL_FUNCTION            2\n",
    "                 12 STORE_NAME               0 (Animal)\n",
    "                 \n",
    " 这时， __build_class__ 函数将默认以 object 为基类创建 Animal 对象。换句话讲，如果自定义类没有显式指定继承关系，将默认继承于 object ，这就是继承链中 object 的由来。\n",
    "\n",
    "当我们创建 Dog 类时，由于代码中明确指定了从 Animal 继承，偏移量为 24 的那条字节码将 Animal 类加载到运行栈并传给 __build_class__ 函数：\n",
    "\n",
    "      5          14 LOAD_BUILD_CLASS\n",
    "                 16 LOAD_CONST               2 (<code object Dog at 0x109bd1c90, file \"\", line 5>)\n",
    "                 18 LOAD_CONST               3 ('Dog')\n",
    "                 20 MAKE_FUNCTION            0\n",
    "                 22 LOAD_CONST               3 ('Dog')\n",
    "                 24 LOAD_NAME                0 (Animal)\n",
    "                 26 CALL_FUNCTION            3\n",
    "                 28 STORE_NAME               1 (Dog)\n",
    "                 \n",
    " 结合对象模型中的知识可知： __build_class__ 函数将基类保存于 PyTypeObject 类型对象的 tp_base 字段中。\n",
    "\n",
    "通过 tp_base 字段，子类与父类被串在一起，形成一条继承链：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_12.jpg\" width=800 height=600>  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aware-amsterdam",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 多继承"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "treated-centre",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在多继承场景，故事又是怎样的呢？我们接着扩展前面的例子：\n",
    "\n",
    "    class Animal:\n",
    "        def run(self):\n",
    "            print('running')\n",
    "\n",
    "    class Dog(Animal):\n",
    "        def yelp(self):\n",
    "            print('woof')\n",
    "        def play(self):\n",
    "            print('playing')\n",
    "\n",
    "    class Sleuth(Dog):\n",
    "        def yelp(self):\n",
    "            print('WOOF!')\n",
    "        def hunt(self):\n",
    "            print('hunting')\n",
    "\n",
    "    class SnifferDog(Dog):\n",
    "        def search(self):\n",
    "            print('searching')\n",
    "\n",
    "    class PoliceDog(Sleuth, SnifferDog):\n",
    "        def patrol(self):\n",
    "            print('patroling')\n",
    "\n",
    "这个例子引入搜救犬类 SnifferDog ，继承于普通狗 Dog ；警犬类 PoliceDog 同时继承于猎犬类 Sleuth 以及搜救犬类 SnifferDog ，表明警犬同时具有猎犬以及搜救犬的特质：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_13.jpg\" width=800 height=600>  \n",
    " \n",
    "接着，我们扒开负责创建 PoliceDog 的字节码围观一下：\n",
    "\n",
    "      21          62 LOAD_BUILD_CLASS\n",
    "                 64 LOAD_CONST               8 (<code object PoliceDog at 0x106b81c00, file \"\", line 21>)\n",
    "                 66 LOAD_CONST               9 ('PoliceDog')\n",
    "                 68 MAKE_FUNCTION            0\n",
    "                 70 LOAD_CONST               9 ('PoliceDog')\n",
    "                 72 LOAD_NAME                2 (Sleuth)\n",
    "                 74 LOAD_NAME                3 (SnifferDog)\n",
    "                 76 CALL_FUNCTION            4\n",
    "                 78 STORE_NAME               4 (PoliceDog)\n",
    "                 80 LOAD_CONST              10 (None)\n",
    "                 82 RETURN_VALUE\n",
    " \n",
    " 注意到，字节码在调用 __build_class__ 函数前，将两个父类 Sleuth 以及 SnifferDog 作为参数按顺序压入栈中。因此，这段字节码等价于：\n",
    "\n",
    "__build_class__(func, 'PoliceDog', Sleuth, SnifferDog)\n",
    "\n",
    "这样一来， __build_class__ 函数的 bases 参数将拿到一个由直接父类组成的元组！\n",
    "\n",
    "在多继承场景，光一个 tp_base 字段不足以保存多个基类， __build_class__ 函数应该将 bases 元组保存在另一个字段中。再次回到 PyTypeObject 源码，不难找出字段 tp_bases 字段，它便保存着基类列表：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_14.jpg\" width=800 height=600>  \n",
    " \n",
    " 在 Python 层可以访问类 __base__ 和 __bases__ 属性，印证了底层字段 tp_base 以及 tp_bases 的作用：\n",
    "\n",
    "    >>> PoliceDog.__base__\n",
    "    <class '__main__.Sleuth'>\n",
    "    >>> PoliceDog.__bases__\n",
    "    (<class '__main__.Sleuth'>, <class '__main__.SnifferDog'>)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "central-bargain",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 属性查找顺序"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorporated-ghana",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们或多或少都知道，子类没有定义的属性，应该沿着继承链到父类中查找：\n",
    "\n",
    "    >>> s = Sleuth()\n",
    "    >>> s.play()\n",
    "    playing\n",
    "    >>> s.run()\n",
    "    running\n",
    "    \n",
    "猎犬类没有定义 play 方法，调用了父类 Dog 中定义的；run 方法也没有定义，则调用了祖类 Animal 中定义的。\n",
    "\n",
    "简单的单继承场景比较好理解，多继承场景就相对复杂一点。我们先来观察 PoliceDog 的行为，看能否得到一些启发。\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_15.jpg\" width=800 height=600>  \n",
    " \n",
    " PoliceDog 类定义了 patrol 方法，PoliceDog 实例便可以调用该方法，这很好理解：\n",
    "\n",
    "    >>> pd = PoliceDog()\n",
    "    >>> pd.patrol()\n",
    "    patroling\n",
    "\n",
    "虽然 PoliceDog 类没有定义 hunt 方法，但其父类 Sleuth 定义了，因此实例也可以调用方法：\n",
    "\n",
    "    >>> pd.hunt()\n",
    "    hunting\n",
    "    \n",
    "虽然父类 Sleuth 以及 祖类 Dog 均定义了 yelp 方法，PoliceDog 实例会选择血缘较近的版本，即 Sleuth 版本：\n",
    "\n",
    "    >>> pd.yelp()\n",
    "    WOOF!\n",
    "    \n",
    "Sleuth 覆盖了其父类 Dog 中的 yelp 方法，这在面向对象中称为 **覆写** ( overwritting )，search 方法也是同理：\n",
    "\n",
    "    >>> pd.search()\n",
    "    searching\n",
    "    \n",
    "至此，我们可以总结出一条非常重要的规则，在属性查找的过程中，子类版本优先级总是高于父类。举个例子， Sleuth 类定义的版本比 Dog 类优先级更高。\n",
    "\n",
    "如果你对常用数据结构比较熟悉的话，可能已猜到其中的玄机：属性查找的顺序就是对类继承关系图的 拓扑排序 ！拓扑排序恰好可以保证有向图中有连接关系节点间的先后顺序。\n",
    "\n",
    "实际上，Python 虚拟机在创建自定义类对象时，对继承关系图进行拓扑排序，并将排序结果保存于 PyTypeObject 中的 tp_mro 字段中，该字段可通过 __mro__ 属性访问：\n",
    "\n",
    "    >>> PoliceDog.__mro__\n",
    "    (<class '__main__.PoliceDog'>, <class '__main__.Sleuth'>, <class '__main__.SnifferDog'>, <class '__main__.Dog'>, <class '__main__.Animal'>, <class 'object'>)\n",
    "    \n",
    "后续对类进行属性查找时，Python 将依照 __mro__ 中保存的顺序逐一查找。这是以空间换时间的又一典型例子，有效避免重复排序导致的计算开销。\n",
    "\n",
    "最后，我们回过头来考察一个更复杂的多继承场景：\n",
    "\n",
    "    class A:\n",
    "        pass\n",
    "\n",
    "    class B(A):\n",
    "        pass\n",
    "\n",
    "    class C(B):\n",
    "        pass\n",
    "\n",
    "    class D(A):\n",
    "        pass\n",
    "\n",
    "    class E(D):\n",
    "        pass\n",
    "\n",
    "    class F(A):\n",
    "        pass\n",
    "\n",
    "    class G(C, E, F):\n",
    "        pass\n",
    "        \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_16.jpg\" width=800 height=600>  \n",
    " \n",
    " 根据继承关系，我们可以得到以下结论：\n",
    "\n",
    "- G 优于 C E F\n",
    "- C 优于 B\n",
    "- E 优于 D\n",
    "- B D F 优于 A\n",
    "- A 优于 object\n",
    "\n",
    "然而有些类之间的先后顺序仅靠继承关系是无法确定的， B E 就是其中的例子。这一点都不意外，有向图的拓扑排序结果可能不止一个。\n",
    "\n",
    "那么，Python 究竟以什么为准呢？\n",
    "\n",
    "    >>> G.__mro__\n",
    "    (<class '__main__.G'>, <class '__main__.C'>, <class '__main__.B'>, <class '__main__.E'>, <class '__main__.D'>, <class '__main__.F'>, <class '__main__.A'>, <class 'object'>)\n",
    "    \n",
    "我们直接看结果，可以等到另外两点结论：\n",
    "\n",
    "- 拓扑排序是深度优先的，遍历 C 之后，继续遍历 B，而不是 E ；\n",
    "- 分支遍历顺序由基类列表定义顺序决定，因此 C E F 一定按照这个顺序遍历；\n",
    "\n",
    "实际上，Python 内部采用 C3算法 ，根据继承关系图生成一个满足以上约束的拓扑排序。如果你对源码比较感兴趣的话，可以从 __build_class__ 函数出发，顺藤摸瓜。印证过程应该不难，因篇幅关系就不过度展开了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stuffed-conservative",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "involved-roads",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- 基类保存于 tp_base 字段，可通过 __base__ 属性访问；\n",
    "- 基类列表保存于 tp_bases 字段，可通过 __bases__ 属性访问；\n",
    "- 类属性查找顺序为类继承图的深度优先 拓扑排序 ；\n",
    "- Python 创建自定义类时，对类继承图进行拓扑排序并保存于 tp_mro 字段，可通过 __mro__ 属性访问；\n",
    "- 查看自定义类 __mro__ 属性，即可确认类属性查找顺序；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "noted-patio",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 属性描述符如何影响属性行为"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "arbitrary-waste",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们先通过一个简单的类 Point ，考察对象属性的行为：\n",
    "\n",
    "    class Point:\n",
    "        z = 0\n",
    "        def __init__(self, x, y):\n",
    "            self.x = x\n",
    "            self.y = y\n",
    "        def distance(self):\n",
    "            return math.sqrt(self.x*self.x + self.y*self.y)\n",
    "            \n",
    "Point 类用于表示一个二维坐标，例如 (1, 2) ：\n",
    "\n",
    "    >>> p = Point(1, 2)\n",
    "    \n",
    "其中， Point 是自定义类对象， p 是 Point 类的实例对象。由于 X 、Y 轴坐标分别作为 x 、y 属性被 __init__ 函数保存于实例对象属性空间中，因而可以通过对象属性查找访问：\n",
    "\n",
    "    >>> 'x' in p.__dict__\n",
    "    True\n",
    "    >>> p.x\n",
    "    1\n",
    "\n",
    "    >>> 'y' in p.__dict__\n",
    "    >>> p.y\n",
    "    2\n",
    "    \n",
    "我们在 Point 类中定义了 distance 方法，用于计算坐标到原点的距离。虽然它并不在 p 实例对象的属性空间中，照样可以被 p 查找到，这又是为什么呢？\n",
    "\n",
    "    >>> 'distance' in p.__dict__\n",
    "    False\n",
    "    >>> p.distance()\n",
    "    2.23606797749979\n",
    "    \n",
    "我们定义的 distance 方法，明明有一个参数 self ，但我们调用时却无须传递！实际上，你可能早已知晓，Python 自动将 p 作为 self 参数传给 distance 。那么，Python 又是如何暗度陈仓的呢？\n",
    "\n",
    "通过观察 p 实例对象 distance 属性，我们找到一些蛛丝马迹：\n",
    "\n",
    "    >>> p.distance\n",
    "    <bound method Point.distance of <__main__.Point object at 0x1077d4160>>\n",
    "    \n",
    "我们惊讶地发现，它居然是一个 bound method 对象，而不是我们定义的函数对象！\n",
    "\n",
    "经过前面章节学习，我们很清楚 distance 方法保存在 Point 类的属性空间，而它只是一个普通的函数对象：\n",
    "\n",
    "    >>> 'distance' in Point.__dict__\n",
    "    True\n",
    "    >>> Point.distance\n",
    "    <function Point.distance at 0x1077b9730>\n",
    "    \n",
    "因此，如果我们通过 Point 类来调用 distance 方法，是需要显式传递 self 参数的：\n",
    "\n",
    "    >>> Point.distance(p)\n",
    "    2.23606797749979\n",
    "    \n",
    "那么，distance 函数对象是如何完成到 bound method 对象的华丽转身的呢？Python 在背后又做了哪些不为人知的事情呢？带着这些疑问，我们继续到 Python 虚拟机中寻找答案。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brazilian-tissue",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 底层布局"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "selective-break",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在对象模型部分，我们知道对 可调用对象 ( callable )进行调用，Python 内部执行其类型对象的 tp_call 函数。\n",
    "\n",
    "由于 Point 类对象是类型对象，其类型是 type 。因此，当我们调用 Point 类创建实例对象，Python 将执行 type 的 tp_call 函数，tp_call 函数则回过头来调用 Point 类的 tp_new 函数。\n",
    "\n",
    "我们并没有给 Point 定义 __new__ 函数，也就是说 Point 类并没有 tp_new 函数。好在 Point 有一个默认基类 object ，而 object 提供了一个兜底的版本：object.tp_new ，它负责为实例对象分配内存。\n",
    "\n",
    "最后，type.tp_call 将调用 Point.tp_init 函数，也就是我们定义的 __init__ 函数，对实例对象进行初始化。\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_17.jpg\" width=800 height=600>  \n",
    " \n",
    " 当 Point 实例对象完成初始化，实例对象与 Point 类对象在底层的内存布局大致是这样的：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_18.jpg\" width=800 height=600>  \n",
    " \n",
    " p 实例对象背后有一个 dict 对象，用于维护其属性空间，实例属性 x 、 y 便藏身其中；同样，Point 类对象背后也有一个 dict 对象，用于维护其属性空间，属性 z 以及方法 distance 也藏身其中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "completed-transparency",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 字节码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "professional-unknown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当我们在 p 实例上调用 distance 方法时，由于 p 实例属性空间并没有定义 distance 方法，Python 将到 Point 类的属性空间中查找。如果 Point 类属性空间也没有定义，Python 继续沿着类继承图逐层向上查找。\n",
    "\n",
    "Python 并没有直接将定义在 Point 类属性空间的 distance 函数对象直接返回，而是对它做了一些手脚。想来这也很合理，如果将 distance 函数对象原样返回的话，我们还要显示传递 self 参数：\n",
    "\n",
    "    p.instance(p)\n",
    "    \n",
    "这可真丑陋！那么，Python 为解决这个问题又做了哪些手脚呢？会不会是由一个特殊的字节码完成的呢？\n",
    "\n",
    "对 p.distance() 这个语句进行反编译，得到以下字节码：\n",
    "\n",
    "    >>> dis.dis(compile('p.distance()', '', 'exec'))\n",
    "      1           0 LOAD_NAME                0 (p)\n",
    "                  2 LOAD_METHOD              1 (distance)\n",
    "                  4 CALL_METHOD              0\n",
    "                  6 POP_TOP\n",
    "                  8 LOAD_CONST               0 (None)\n",
    "                 10 RETURN_VALUE\n",
    "                 \n",
    "接着对 p.distance 这个语句进行反编译，得到以下字节码：\n",
    "\n",
    "    >>> dis.dis(compile('p.distance', '', 'exec'))\n",
    "      1           0 LOAD_NAME                0 (p)\n",
    "                  2 LOAD_ATTR                1 (distance)\n",
    "                  4 POP_TOP\n",
    "                  6 LOAD_CONST               0 (None)\n",
    "                  8 RETURN_VALUE\n",
    "                  \n",
    "我们发现了一个全新的字节码 LOAD_METHOD ，用于查找并加载对象方法！秘密会不会就在这个字节码上？等等！我们刚提过，通过普通属性查找得到的 distance 方法，也是加工过的：\n",
    "  \n",
    "      >>> p.distance\n",
    "    <bound method Point.distance of <__main__.Point object at 0x1077d4160>>\n",
    "\n",
    "这样看来，LOAD_ATTR 字节码也会对返回的属性做手脚，因此答案跟 LOAD_METHOD 字节码应该没有必然联系。虽然白忙活了一场，但我们已经找到了解开秘密的钥匙——只需扒一扒这两个字节码的处理逻辑即可。\n",
    "\n",
    "我们从 LOAD_ATTR 入手，借助它一步步接近幕后的无名英雄——Python 描述符 ( descriptor )。过程因篇幅关系省略一万字，有兴趣的童鞋请参考该字节码相关处理逻辑，源码位于 Python/ceval.c 文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "designed-chicago",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 描述符"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controlling-tooth",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们发现，函数对象还有不为人知的一面，它同时也是一个描述符。那么，什么是描述符呢？\n",
    "\n",
    "Python 描述符是一个包含“绑定行为”的对象，对描述符的属性访问受描述符协议定义的方法影响，这样的方法有：\n",
    "\n",
    "- __get__() ，影响属性获取；\n",
    "- __set__() ，影响属性设置；\n",
    "- __delete__() ，影响属性删除；\n",
    "\n",
    "由于函数对象定义了 __get__() 方法，因此函数对象是一个描述符对象：\n",
    "\n",
    "    >>> def foo():\n",
    "    ...     pass\n",
    "    ...\n",
    "    >>> hasattr(foo, '__get__')\n",
    "    True\n",
    "    >>> hasattr(Point.distance, '__get__')\n",
    "    True\n",
    "    \n",
    "接下来，我们进一步考察描述符是如何影响属性行为的。我们知道，Python 访问对象属性时，默认先在对象的属性空间字典中查找；再到类型对象的属性空间查找。\n",
    "\n",
    "如果属性可在对象属性空间字典中找到，Python 原样返回，不加修饰。举个例子，我们将 distance 函数对象做为 distance1 属性保存到实例对象 p 的属性空间中：\n",
    "\n",
    "    >>> Point.distance\n",
    "    <function Point.distance at 0x10dace6a8>\n",
    "    >>> p.distance1 = Point.distance\n",
    "    >>> 'distance1' in p.__dict__\n",
    "    True\n",
    "    \n",
    "我们访问 distance1 属性，获得了 distance 函数对象，未经任何修饰：\n",
    "\n",
    "    >>> p.distance1\n",
    "    <function Point.distance at 0x10dace6a8>\n",
    "    >>> p.distance1(p)\n",
    "    2.23606797749979\n",
    "    \n",
    "    \n",
    "如果属性在对象属性空间中找不到，Python 接着在其类型对象属性空间中查找，按照上节介绍的顺序回溯继承图。这时，需要分成两种情况来讨论：\n",
    "\n",
    "如果在类型对象属性空间中找到的属性不是描述符，Python 原样返回。举个例子，通过 p 访问类属性 z ：\n",
    "\n",
    "    >>> p.z\n",
    "    0\n",
    "    \n",
    "如果在类型对象属性空间中找到的属性是描述符，Python 将调用其 __get__ 方法，将实例对象与它进行绑定。\n",
    "\n",
    "同样以 p 调用 distance 方法为例，Python 在 Point 类属性空间中找到 distance 函数对象，发现它是一个描述符。接着，Python 调用 distance 函数的 __get__ 方法，并将 p 作为参数传进去，以此与 p 完成绑定：\n",
    "\n",
    "    >>> Point.distance\n",
    "    <function Point.distance at 0x10087a730>\n",
    "    >>> Point.distance.__get__(p)\n",
    "    <bound method Point.distance of <__main__.Point object at 0x100896630>>\n",
    "    \n",
    "这样一来，我们对 distance 进行属性查找得到的是一个 bound method ，它的底层结构如下：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_19.jpg\" width=800 height=600>  \n",
    " \n",
    " bound method 底层是一个 PyMethodObject 结构体，定义于 Include/classobject.h 头文件中。该结构体除了一些公共头部，只有两个重要字段：\n",
    "\n",
    "- im_func ，函数对象；\n",
    "- im_self ，绑定的实例对象；\n",
    "\n",
    "所有迷雾都拨开了，p 实例对象就是这样与 distance 函数对象紧紧地绑定在一起的！\n",
    "\n",
    "    >>> bm = p.distance\n",
    "    >>> bm\n",
    "    <bound method Point.distance of <__main__.Point object at 0x100896630>>\n",
    "\n",
    "    >>> bm.__func__\n",
    "    <function Point.distance at 0x10087a730>\n",
    "    >>> bm.__func__ is Point.distance\n",
    "    True\n",
    "\n",
    "    >>> bm.__self__\n",
    "    <__main__.Point object at 0x100896630>\n",
    "    >>> bm.__self is p\n",
    "    True\n",
    "    \n",
    "当我们调用 bound method 时，Python 将取出 im_func 函数，然后以 im_self 为参数调用它。因此，下面两种不同的调用方式是等价的：\n",
    "\n",
    "    >>> bm()\n",
    "    2.23606797749979\n",
    "    >>> bm.__func__(bm.__self__)\n",
    "    2.23606797749979\n",
    "    \n",
    "__get__ 方法只影响属性获取的行为，也就是说它是一种只读行为。而属性还可以被修改和删除，只是可以通过 __set__ 以及 __delete__ 施加影响。根据这几个方法，描述符还可进一步分成：\n",
    "\n",
    "- 非数据描述符，只实现 __get__ 方法；\n",
    "- 数据描述符，定义了 __set__ 或者 __delete__ 方法；\n",
    "\n",
    "显然，函数对象是一种非数据描述符。至于数据描述符，因篇幅关系先按下不表，留在魔术方法一节再行展开。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suffering-fundamental",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 高阶编程：魔术方法应用实战"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worthy-exchange",
   "metadata": {
    "hidden": true
   },
   "source": [
    "魔术方法是什么呢？顾名思义，魔术方法是一种特殊的方法，可以让自定义类具有某种魔法。魔术方法名开头结尾都包含两个下划线，例如 __init__ 方法，负责对实例对象进行初始化。\n",
    "\n",
    "下表将常用的魔术方法分门别类，你或多或少可能已有所了解：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_20.jpg\" width=800 height=600>  \n",
    "\n",
    "合理应用魔术方法，让自定义类更具 Python 格调，更好地践行 Python 数据抽象以及设计哲学，是每个 Python 工程师必备的编程技巧。接下来，我们一起来考察几个典型的案例，以此抛砖引玉。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driving-thanks",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 运算符重载"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "boolean-appliance",
   "metadata": {
    "hidden": true
   },
   "source": [
    "运算符，诸如加减乘除 ( +-*/ )，处理逻辑由 __add__ 等数值操作魔术方法控制。以加法运算符 + 为例，表达式 a + b 在 Python 内部是这样求值的：\n",
    "\n",
    "1. 如果 a 定义了魔术方法 __add__ ，则调用 a.__add__(b) 进行求值；\n",
    "2. 如果 b 定义了魔术方法 __radd__ ，则调用 b.__radd__(a) 进行求值；\n",
    "因此，只需要提供相关魔术方法，非数值型对象也可以支持算术运算符。\n",
    "\n",
    "举个例子， str 对象以加法进行对象拼接，以乘法进行对象重复，而除法却没有定义：\n",
    "\n",
    "    >>> 'hello' + ' ' + 'world'\n",
    "    'hello world'\n",
    "    >>> 'abc' * 10\n",
    "    'abcabcabcabcabcabcabcabcabcabc'\n",
    "    >>> 'hello world' / ' '\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    TypeError: unsupported operand type(s) for /: 'str' and 'str'\n",
    "    \n",
    "字符串切分是一个比较常见的操作，若能借助除法操作符来进行则方便许多。实现这个目标并不难，我们只需为 str 编写一个派生类，并实现魔术方法 __truediv__ 即可：\n",
    "\n",
    "    class SmartString(str):\n",
    "\n",
    "        def __truediv__(self, other):\n",
    "            return self.split(other)\n",
    "    \n",
    "在 Python 内部，除法操作 / 由 __truediv__ 魔术方法处理。请注意，在老版本 Python 中，除法操作符由 __div__ 魔术方法处理，名字略有差异。\n",
    "\n",
    "就这么几行代码，我们的字符串类便支持通过除法操作进行字符串切分了：\n",
    "\n",
    "    >>> s = SmartString('hello world')\n",
    "    >>> s / ' '\n",
    "    ['hello', 'world']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "excessive-radical",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 数值型运算"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solid-genre",
   "metadata": {
    "hidden": true
   },
   "source": [
    "结构稍微复杂一些的类型，也可以通过实现数值型魔术函数，让它具备数值类型的行为，支持常见的算术操作。\n",
    "\n",
    "举个简单的例子，我们可以实现一个类来表示向量，x、 y 属性分别存储向量的坐标：\n",
    "\n",
    "    class Vector:\n",
    "        def __init__(self, x, y):\n",
    "            self.x = x\n",
    "            self.y = y\n",
    "        def __add__(self, other):\n",
    "            return Vector(x=self.x+other.x, y=self.y+other.y)\n",
    "        def __sub__(self, other):\n",
    "            return Vector(x=self.x-other.x, y=self.y-other.y)\n",
    "        def __mul__(self, scale):\n",
    "            return Vector(x=self.x*scale, y=self.y*scale)\n",
    "        def __truediv__(self, scale):\n",
    "            return Vector(x=self.x/scale, y=self.y/scale)\n",
    "        def __repr__(self):\n",
    "            return 'Vector(x={}, y={})'.format(self.x, self.y)\n",
    "                   \n",
    "- __add__ 魔术方法，实现向量加法；\n",
    "- __sub__ 魔术方法，实现向量减法；\n",
    "- __mul__ 魔术方法，实现向量数乘；\n",
    "- __truediv__ 魔术方法，实现向量数除；\n",
    "- __repr__ 魔术方法，实现向量表示；\n",
    "\n",
    "这样一来，我们就可以通过常用算术运算符进行向量运算：\n",
    "\n",
    "    >>> v1 = Vector(1, 2)\n",
    "    >>> v2 = Vector(3, 4)\n",
    "    >>> v1 + v2\n",
    "    Vector(x=4, y=6)\n",
    "    >>> v1 * 3\n",
    "    Vector(x=3, y=6)\n",
    "    >>> v1 / 2\n",
    "    Vector(x=0.5, y=1.0)\n",
    "    \n",
    "将自定义类与 Python 的运行哲学相融合，还可带来一些额外的收益 —— 充分发挥 Python 的强大执行能力。\n",
    "\n",
    "举个例子，我们可以借助现成的 sum 内建函数对多个向量进行求和，完全不需要任何额外的代码：\n",
    "\n",
    "    >>> v1 = Vector(1, 2)\n",
    "    >>> v2 = Vector(3, 4)\n",
    "    >>> v3 = Vector(10, 2)\n",
    "    >>> sum((v1, v2, v3), Vector(0, 0))\n",
    "    Vector(x=14, y=8)\n",
    "    \n",
    "看，Python 语言的表达能力就是这样强大！只需对 Python 设计哲学以及相关运行时约定稍有了解，即可将这一切发挥得淋漓尽致！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "least-airfare",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 属性描述符"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dependent-hamilton",
   "metadata": {
    "hidden": true
   },
   "source": [
    "上一小节，我们考察了属性描述符，知道它控制着属性查找的行为。属性描述符分为两种：\n",
    "\n",
    "- **非数据描述符** ：只实现了 __get__ 方法；\n",
    "- **数据描述符** ：至少实现了 __set__ 或者 __delete__ 方法；\n",
    "我们对非数据描述符已经有所了解：函数对象就是其中最典型的一个，它包含着类方法 self 参数绑定的全部秘密。然而，我们对数据描述符却一无所知！别急，我们将通过一个典型的例子，将这部分知识补齐。\n",
    "\n",
    "对于对象 o ，其类型对象为 t 。如果 t 包含数据描述符属性 a ，那么属性设置操作 o.a = x 被 a.__set__ 方法接管；同理，属性删除操作 del o.a 则被 a.__delete__ 方法接管。\n",
    "\n",
    "如果对象 o 属性空间也存在属性 a ，到底以谁为准呢？简而言之，Python 将照以下优先级逐一确定：\n",
    "\n",
    "1. 数据描述符：如果类型对象 (含父类) 定义了同名数据描述符属性，属性操作将被其接管；\n",
    "2. 对象属性：除了①，属性操作默认在属性空间中完成；\n",
    "3. 非数据描述符：属性访问时，如果①②均不成功，而类型对象 (含父类) 定义了同名非数据描述符，属性访问将被其接管；\n",
    "\n",
    "因此，数据描述符优先级最高，对象属性空间次之，非数据描述符最低。下图是一个典型的例子：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_21.jpg\" width=800 height=600>  \n",
    "\n",
    "- 对于属性 a ，由于类型对象 t 属性空间定义了数据描述符，将屏蔽实例对象 o 属性空间中的定义；\n",
    "- 对于属性 b ，由于类型对象 t 属性空间定义的只是非数据描述符，仍以实例对象 o 属性空间定义的为准；\n",
    "- 对于属性 c ，由于实例对象 o 属性空间未定义，属性访问将以类型对象 t 属性空间定义的非数据描述符为准；\n",
    "- 对于属性 c ，由于类型对象 t 属性空间定义的只是非数据描述符，属性设置、删除仍以实例对象 o 属性空间为准；\n",
    "\n",
    "那么，数据描述符到底有什么用处呢？最典型的场景是利用它来实现针对属性操作的统一验证。\n",
    "\n",
    "假设我们有一个用户类 User ，其中 email 属性表示用户邮箱地址：\n",
    "\n",
    "    class User:\n",
    "\n",
    "        def __init__(self, name, email):\n",
    "            self.name = name\n",
    "            self.email = email\n",
    "\n",
    "检查用户邮箱地址合法性对应用逻辑非常重要，这样的赋值不应该被允许：\n",
    "\n",
    "\n",
    "    user = User('jim', 'not an email')\n",
    " \n",
    "我们可以在赋值前利用正则表达式对邮箱进行检查：\n",
    " \n",
    "    import re\n",
    "\n",
    "    class User:\n",
    "\n",
    "        EMAIL_MOD = re.compile('^[a-z0-9]+[\\._]?[a-z0-9]+[@]\\w+[.]\\w{2,3}$')\n",
    "\n",
    "    def __init__(self, name, email):\n",
    "        self.name = name\n",
    "\n",
    "        if not self.EMAIL_MOD.match(email):\n",
    "            raise Exception('{} is not a valid email'.format(email))\n",
    "\n",
    "        self.email = email    \n",
    "        \n",
    "但这种实现方式无法处理一般化的属性设置，漏网之鱼难以避免：\n",
    "\n",
    "    user.email = 'not an email'\n",
    "\n",
    "我们可以在属性设置前加上检查逻辑，但难免会有错漏，代码也略显累赘：\n",
    "\n",
    "    if not self.EMAIL_MOD.match(value):\n",
    "        raise Exception('{} is not a valid email'.format(value))\n",
    "    user.email = 'not an email'\n",
    "    \n",
    "借助 __setattr__ 魔术函数，我们可以对属性设置进行特殊处理：\n",
    "\n",
    "    import re\n",
    "\n",
    "    class User:\n",
    "\n",
    "        EMAIL_MOD = re.compile('^[a-z0-9]+[\\._]?[a-z0-9]+[@]\\w+[.]\\w{2,3}$')\n",
    "\n",
    "        def __init__(self, name, email):\n",
    "            self.name = name\n",
    "            self.email = email\n",
    "\n",
    "        def __setattr__(self, name, value):\n",
    "            if name == 'email':\n",
    "                if not self.EMAIL_MOD.match(value):\n",
    "                    raise Exception('{} is not a valid email'.format(value))\n",
    "                super(User, self).__setattr__(name, value)\n",
    "                return\n",
    "\n",
    "            super(User, self).__setattr__(name, value)  \n",
    "            \n",
    "由于所有属性设置都会被 __setattr__ 接管，因此我们先判断属性名 name ，仅对 email 属性进行前置检查。检查如果通过，则通过 super 调用父类的 __setattr__ 完成属性设置。\n",
    "\n",
    "这样一来，所有对 email 属性的赋值，都会进行合法性检查：\n",
    "\n",
    "    >>> user.email = 'not an email'\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"<stdin>\", line 9, in __setattr__\n",
    "    Exception: not an email is not a valid email\n",
    "    \n",
    "美中不足的是，不仅 email 属性，其他所有属性设置都会被 __setattr__ 接管！如果还需要检查其他属性，势必又要在 __setattr__ 中新增 if 判断。大量代码杂糅在一起，显然不够优雅。\n",
    "\n",
    "铺垫了一大圈，杀手锏上场了！我们可以为 email 属性实现一个数据描述符：\n",
    "\n",
    "    class EmailField:\n",
    "\n",
    "        EMAIL_MOD = re.compile('^[a-z0-9]+[\\._]?[a-z0-9]+[@]\\w+[.]\\w{2,3}$')\n",
    "\n",
    "        def __init__(self, attrname):\n",
    "            self.attrname = attrname\n",
    "\n",
    "        def __get__(self, instance, owner):\n",
    "            return getattr(instance, self.attrname)\n",
    "\n",
    "        def __set__(self, instance, value):\n",
    "            if not self.EMAIL_MOD.match(value):\n",
    "                raise Exception('{} is not a valid email'.format(value))\n",
    "\n",
    "            setattr(instance, self.attrname, value)\n",
    "\n",
    "        def __delete__(self, instance):\n",
    "            raise Exception('can not delete email')\n",
    "            \n",
    "邮箱仍然需要保存在实例对象属性空间中，但属性名不能用 email 了，可以是 _email 或者其他。为此，这个例子将底层属性名作为自定义参数 attrname 保存在描述符中。\n",
    "\n",
    "我们接着考察这个描述符几个关键魔术方法的实现：\n",
    "\n",
    "- __get__ 方法：参数 instance 代表实例对象，owner 代表描述符所在的类对象，方法直接从实例对象中取出底层属性并返回；\n",
    "- __set__ 方法：参数 instance 代表实例对象，value 表示新属性值，方法先检查属性值是否是一个合法邮箱，在调用 setattr 函数将其作为底层属性保存在实例对象中；\n",
    "- __delete__ 方法，直接抛异常，禁止删除邮箱；\n",
    "\n",
    "借助 EmailField 描述符，我们的 User 类可以这样定义：\n",
    "\n",
    "    class User:\n",
    "\n",
    "        email = EmailField(attrname='_email')\n",
    "\n",
    "        def __init__(self, name, email):\n",
    "            self.name = name\n",
    "            self.email = email\n",
    "\n",
    "        def __repr__(self):\n",
    "            return \"User(name='{}', email='{}')\".format(self.name, self.email)\n",
    "            \n",
    "注意到，我们选择 _email 作为底层属性。接着，我们创建一个 User 实例对象 user ：\n",
    "\n",
    "    >>> user\n",
    "    User(name='fasionchan', email='admin@fasionchan.com')\n",
    "    >>> user.email\n",
    "    'admin@fasionchan.com'\n",
    "\n",
    "当我们更新 email 属性时，非法邮箱将无法通过：\n",
    "\n",
    "    >>> user.email = 'not an email'\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"<stdin>\", line 9, in __set__\n",
    "    Exception: not an email is not a valid email\n",
    "    \n",
    "当然了，我们已经无法将邮箱删除了，如预期的那样：\n",
    "\n",
    "    >>> del user.email\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"<stdin>\", line 12, in __delete__\n",
    "    Exception: can not delete email\n",
    "    \n",
    "回过头来看一下实例对象 user 的属性空间，注意到邮箱地址是以 _email 属性保存的：\n",
    "\n",
    "    >>> user.__dict__\n",
    "    {'name': 'fasionchan', '_email': 'admin@fasionchan.com'}\n",
    "\n",
    "顺便提一下，在这个例子中 __get__ 函数 instance 参数为 user 实例对象，而 owner 参数为 User 类对象。\n",
    "\n",
    "有了数据描述符协议，我们还可以设计出更加巧妙的数据模式，不少数据库 ORM 框架也是这样实现的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "downtown-siemens",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 上下文管理器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "athletic-ozone",
   "metadata": {
    "hidden": true
   },
   "source": [
    "程序有时需要向系统申请一些资源，使用后必须释放，否则将导致资源泄露。\n",
    "\n",
    "文件是一个典型的例子：当进程向操作系统申请打开一个文件时，操作系统将分配一个文件描述符 (或者叫做文件句柄)；通过文件描述符，进程可以对文件进行读写操作；当所有操作完成后，进程必须将文件关闭，以便释放文件描述符：\n",
    "\n",
    "    f = open('some_file')\n",
    "\n",
    "    for line in f:\n",
    "        process(line)\n",
    "\n",
    "    f.close()\n",
    "    \n",
    "然而，当我们在操作文件时，程序可能会抛出异常，打破正常的执行逻辑。如上述例子，如果 process 函数抛出异常，f.close() 这个语句便永远不会执行了。这时，文件描述符泄露也就无法避免了。\n",
    "\n",
    "因此，我们需要通过 try finally 结构，确保 f.close() 不受异常影响，一定执行\n",
    "\n",
    "        f = open('some_file')\n",
    "\n",
    "    try:\n",
    "        for line in f:\n",
    "            process(line)\n",
    "    finally:\n",
    "        f.close()\n",
    "        \n",
    "try finally 结构的引入，完美地解决了问题。美中不足的是，代码结构看上去不够优雅，特别是当代码层级较深时。\n",
    "\n",
    "这时，我们可以借助 with 关键字，做进一步完善：\n",
    "\n",
    "    with open('some_file') as f:\n",
    "        for line in f:\n",
    "            process(f)\n",
    "                      \n",
    "这个例子充分利用文件对象作为 **上下文管理器** ( Context Manager ) 的特性：当程序离开 with 代码块时，将自动关闭文件，不管是正常执行完毕还是抛异常。\n",
    "\n",
    "上下文管理器一般与 with 关键字搭配使用，可精确控制资源的分配与回收。上下文管理器需要提供两个魔术方法：\n",
    "\n",
    "- __enter__ ，程序在进入 with 代码块前执行，一般用来分配资源，返回值将赋值给 as 关键字指定的变量；\n",
    "- __exit__ ，程序离开 with 代码块后执行，一般用来回收资源；\n",
    "\n",
    "一个类只要实现了 __enter__ 和 __exit__ 魔术方法，便是一个合法的上下文管理器。借此，我们可以优雅地实现一些有意思的功能。举个例子，实现一个计时器，用于跟踪代码执行时间：\n",
    "\n",
    "    import time\n",
    "\n",
    "    class CodeTimer:\n",
    "        def __init__(self, name):\n",
    "            self.name = name\n",
    "        def __enter__(self):\n",
    "            self.enter_time = time.time()\n",
    "            print('code timer started, name={}'.format(self.name))\n",
    "            return self\n",
    "        def __exit__(self, type, value, traceback):\n",
    "            print('code timer stop, name={}, expired={}'.format(\n",
    "                self.name,\n",
    "                time.time()-self.enter_time,\n",
    "            ))\n",
    "            \n",
    "CodeTimer 类很好理解，它只有两个关键方法：\n",
    "\n",
    "- __enter__ ，记录当前时间并输出启动提示；\n",
    "- __exit__ ，以当前时间减去启动时间计算代码执行时长并输出到屏幕；\n",
    "\n",
    "好了，现在可以用 CodeTimer 来追踪代码执行时间：\n",
    "\n",
    "    >>> with CodeTimer('example-1'):\n",
    "    ...     print('code block starting')\n",
    "    ...     time.sleep(1)\n",
    "    ...     print('code block ending')\n",
    "    ...\n",
    "    code timer started, name=example-1\n",
    "    code block starting\n",
    "    code block ending\n",
    "    code timer stop, name=example-1, expired=1.0018329620361328\n",
    "    \n",
    "请留意程序输出内容，这表明 __enter__ 在代码块开始执行前便执行，而 __exit__ 在代码块执行完毕后才执行。  \n",
    "\n",
    "接着，我们把 0 作为除数，让代码块抛异常。尽管如此， CodeTimer 还是成功输出代码块执行耗时：\n",
    "\n",
    "    >>> with CodeTimer('example-1'):\n",
    "    ...     time.sleep(1)\n",
    "    ...     1 / 0\n",
    "    ...     time.sleep(1)\n",
    "    ...\n",
    "    code timer started, name=example-1\n",
    "    code timer stop, name=example-1, expired=1.0011489391326904\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 3, in <module>\n",
    "    ZeroDivisionError: division by zero\n",
    "    \n",
    "通过上下文管理器协议，我们将代码执行计时的细节封装在 CodeTimer 内部。这不仅增强了程序的健壮性，还提高了代码的复用能力，一举多得。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceramic-month",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## metaclass 在程序开发中的妙用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hungarian-protection",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 类型系统中，元类 ( metaclass )占据着至关重要的位置。它提供了制造新类型的能力，为程序设计带来更多可能性。不少功能强大的开发框架，内部实现离不开元类的强力加持。\n",
    "\n",
    "然而，元类由于天生的抽象性，让很多人望而却步。本节我们以若干应用场景抛砖引玉，带你拿下元类这个新技能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clear-simon",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### __new__ 魔术方法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civil-integrity",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们先考察 __new__ 魔术方法，它负责为类创建实例对象。什么？实例对象不是由 __init__ 魔术方法创建吗？为了拨开心中的迷雾，我们先将这两者的区别和联系搞清楚。\n",
    "\n",
    "当我们调用类 C 创建实例对象时，Python 调用 C.__new__ 魔法函数完成对象创建：\n",
    "\n",
    "    instance = C.__new__()\n",
    "    \n",
    "如果 C 类并没有定义 __new__ 方法，Python 将使用父类的 __new__ 方法。如果父类也没有实现该方法，Python 将逐层上溯，直到终极父类 object ，而 object 提供了通用的 __new__ 版本进行兜底。\n",
    "\n",
    "将实例对象 instance 返回给调用者之前，C.__new__ 调用 instance.__init__ 魔术方法对实例对象进行初始化：\n",
    "\n",
    "    instance.__init__()\n",
    "    \n",
    "- __new__ ，负责为创建实例对象(分配内存)；\n",
    "- __init__ ，负责对实例对象进行初始化；\n",
    "\n",
    "由于 __init__ 魔术方法执行时，实例对象 self 已经创建好了，因此无法对创建过程施加影响。另一方面，__new__ 魔术方法刚执行时，实例对象还未诞生，因此它不可能以实例方法的形式存在，只能以类方法的形式存在。\n",
    "\n",
    "那么，有什么场景需要应用 __new__ 魔术方法呢？——最典型的例子是：**单例模式** 的实现。\n",
    "\n",
    "举个例子，假设我们有一个计数器类 Counter 用于统计应用访问量：\n",
    "\n",
    "    class Counter:\n",
    "\n",
    "        def __init__(self):\n",
    "            self.value = 0\n",
    "            self.lock = Lock()\n",
    "\n",
    "        def inc(self):\n",
    "            with self.lock:\n",
    "                self.value += 1\n",
    "\n",
    "在程序很多地方都会调用计数器的 inc 方法，对计数器进行自增：\n",
    "\n",
    "    counter = Counter()\n",
    "    counter.inc()\n",
    "    \n",
    "由于计数器全局只需一个，我们希望把它做成单例。换句话讲，不管我们调用 Counter 多少次，它都返回全局唯一的一个 Counter 实例：\n",
    "\n",
    "    >>> Counter() is Counter()\n",
    "       True\n",
    "\n",
    "因此，当 Counter 创建第一个实例后，我们需要将它记录起来；后续再调用 Counter ，直接将其返回。结合 __new__ 和 __init__ 这两种魔术方法的运行机制，我们知道应该在 __new__ 方法中做文章：\n",
    "\n",
    "    class Counter:\n",
    "\n",
    "        instance = None\n",
    "\n",
    "        def __new__(cls, *args, **kwargs):\n",
    "            if cls.instance is None:\n",
    "                cls.instance = super().__new__(cls, *args, **kwargs)\n",
    "\n",
    "            return cls.instance\n",
    "\n",
    "        def __init__(self):\n",
    "            self.value = 0\n",
    "            self.lock = Lock()\n",
    "\n",
    "        def inc(self):\n",
    "            with self.lock:\n",
    "                self.value += 1\n",
    "                \n",
    "我们引入类属性 instance ，用于记录唯一的实例对象。在 __new__ 方法中，我们先检查 instance 属性：如果发现它尚未创建，则调用父类 __new__ 方法进行创建；否则，直接将实例返回。\n",
    "\n",
    "如果存在多个线程并发调用 Counter 创建实例对象的情况，将产生 竞争态 。举个例子，假设两个线程同时检查了 instance 属性，发现实例对象尚未创建，然后分头创建实例对象。\n",
    "\n",
    "解决竞争态的手段也很简单，只需为 __new__ 函数也加上一把锁，就像 inc 方法一样：\n",
    "\n",
    "    class Counter:\n",
    "\n",
    "        lock = Lock()\n",
    "        instance = None\n",
    "\n",
    "        def __new__(cls, *args, **kwargs):\n",
    "            with cls.lock:\n",
    "                if cls.instance is None:\n",
    "                    cls.instance = super().__new__(cls, *args, **kwargs)\n",
    "\n",
    "                return cls.instance\n",
    "\n",
    "        def __init__(self):\n",
    "            self.value = 0\n",
    "\n",
    "        def inc(self):\n",
    "            with self.lock:\n",
    "                self.value += 1\n",
    "                \n",
    "注意到，例子将锁从实例属性，调整成类属性。这样一来，一把锁即可同时保护类方法( __new__ )和实例方法( inc )。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "specialized-federal",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### metaclass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inside-demonstration",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在 Python 对象模型中，类型对象的类型是一个特殊的类型对象，称为 **元类** ( metaclass )。元类可以看作一个类型工厂，可以制造新的类型对象。\n",
    "\n",
    "type 对象我们已经非常熟悉，它是 Python 内置的元类，提供了制造新类型的最基本能力。我们编写自定义类时，正是它在背后默默工作。\n",
    "\n",
    "元类引入了一种强大的魔力，但它的 **抽象性** 却令许多人为之却步。为此，我准备了一个非常典型的例子，力求将元类的原理一次性讲透。为尽量降低阅读难度，我对例子作了最大程度的简化。\n",
    "\n",
    "假设我们正在设计了一个用于实现插件的基类，它提供了 serve_forever 方法，具备根据指定时间间隔，循环执行 process 处理函数的基础能力：   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "shaped-employee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:35:08.524862Z",
     "start_time": "2021-03-30T04:35:08.513883Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "class BasePlugin:\n",
    "\n",
    "    def __init__(self, interval):\n",
    "        self.interval = interval\n",
    "\n",
    "    def serve_forever(self):\n",
    "        while True:\n",
    "            self.process()\n",
    "            time.sleep(self.interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulated-fantasy",
   "metadata": {
    "hidden": true
   },
   "source": [
    " 当 Python 执行这段代码时，最终调用 type 对象完成 BasePlugin 类对象的创建，伪代码大致是这样的：\n",
    " \n",
    "     attrs = {\n",
    "        '__init__': <function>,\n",
    "        'serve_forever': <function>,\n",
    "    }\n",
    "\n",
    "    BasePlugin = type(name='BasePlugin', bases=(object,), attrs=attrs)\n",
    "    \n",
    "实际插件作为 BasePlugin 的子类来组织，子类从基类继承了 serve_forever 的能力，并在 process 方法中实现具体处理逻辑："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "spatial-reply",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:35:11.219567Z",
     "start_time": "2021-03-30T04:35:11.212573Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class BarPlugin(BasePlugin):\n",
    "\n",
    "    def process(self):\n",
    "        print('bar processing')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "planned-stress",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这时，确保子类 BarPlugin 实现了 process 方法就显得格外重要，只有满足这一点才算是一个合法的插件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "expanded-radiation",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:35:13.088974Z",
     "start_time": "2021-03-30T04:35:13.078985Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class FaultPlugin(BasePlugin):\n",
    "\n",
    "    def run(self):\n",
    "        print('xxxx')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "formal-belize",
   "metadata": {
    "hidden": true
   },
   "source": [
    "FaultPlugin 就不是一个合法的插件，它没有实现 process 方法。然而，Python 默认无法发现这一点。\n",
    "\n",
    "那么，有没有办法让 Python 在创建新类时，自动检查它是否实现了某个方法呢？我们接着探索。\n",
    "\n",
    "理所当然，FaultPlugin 类是 type 的实例对象，默认也是 type 创建的：\n",
    "\n",
    "    attrs = {\n",
    "        'run': <function>\n",
    "    }\n",
    "\n",
    "    FaultPlugin = type(name='FaultPlugin', bases=(BasePlugin,), attrs=attrs)\n",
    "    \n",
    "但 type 作为一个通用的元类，并不会帮我们检查 attrs 属性空间，确保 process 函数存在。\n",
    "\n",
    "理论上，只要继承 type 来实现新元类，并在子类中重写 type 的默认行为，便可实现检查 process 函数的目的。那么，有办法为自定义类指定元类吗？答案是肯定的——我们可以在类定义中通过 metaclass 指定。\n",
    "\n",
    "假设我们实现的新元类名为 PluginMeta ，而 BasePlugin 通过 metaclass 参数指定以它为元类："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "angry-video",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:35:32.467746Z",
     "start_time": "2021-03-30T04:35:32.448744Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class BasePlugin(metaclass=PluginMeta):\n",
    "    \n",
    "    def __init__(self, interval):\n",
    "        self.interval = interval\n",
    "        \n",
    "    def serve_forever(self):\n",
    "        while True:\n",
    "            self.process()\n",
    "            time.sleep(self.interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exceptional-bottle",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这意味着，Python 在创建 BasePlugin 类及子类对象时，将调用 PluginMeta ，而不是直接调用 type 了：\n",
    "\n",
    "    attrs = {\n",
    "        '__init__': <function>,\n",
    "        'serve_forever': <function>,\n",
    "    }\n",
    "\n",
    "    BasePlugin = PluginMeta(name='BasePlugin', bases=(object,), attrs=attrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strange-pollution",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于实例对象是 __new__ 魔术方法负责创建的，因此我们需要覆写 type 类的 __new__ 方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "focused-canon",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:35:31.892186Z",
     "start_time": "2021-03-30T04:35:31.877181Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class PluginMeta(type):\n",
    "\n",
    "    def __new__(cls, name, bases, dct):\n",
    "        if name != 'BasePlugin' and 'process' not in dct:\n",
    "            raise TypeError('subclass {} should implement process method'.format(name))\n",
    "        return super().__new__(cls, name, bases, dct)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cellular-indonesian",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在子类 __new__ 方法，我们先检查属性空间字典。如果发现 process 方法不存在，则直接抛出 TypeError 异常。注意到，基类 BasePlugin 没实现 process 是正常的，我们通过类名判断，对它豁免检查。\n",
    "\n",
    "现在，Python 就像有了火眼金睛一样，能够轻松识别非法插件子类—— FaultPlugin ：\n",
    "\n",
    "    TypeError: subclass FaultPlugin should implement process method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "lasting-compromise",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:36:57.722661Z",
     "start_time": "2021-03-30T04:36:57.704662Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "subclass FaultPlugin should implement process method",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mTypeError\u001B[0m                                 Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-8-e63b57e5b013>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[1;32mclass\u001B[0m \u001B[0mFaultPlugin\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mBasePlugin\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m      2\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      3\u001B[0m     \u001B[1;32mdef\u001B[0m \u001B[0mrun\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mself\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      4\u001B[0m         \u001B[0mprint\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'xxxx'\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;32m<ipython-input-6-e12df00d48ab>\u001B[0m in \u001B[0;36m__new__\u001B[1;34m(cls, name, bases, dct)\u001B[0m\n\u001B[0;32m      3\u001B[0m     \u001B[1;32mdef\u001B[0m \u001B[0m__new__\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mcls\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mname\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mbases\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      4\u001B[0m         \u001B[1;32mif\u001B[0m \u001B[0mname\u001B[0m \u001B[1;33m!=\u001B[0m \u001B[1;34m'BasePlugin'\u001B[0m \u001B[1;32mand\u001B[0m \u001B[1;34m'process'\u001B[0m \u001B[1;32mnot\u001B[0m \u001B[1;32min\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 5\u001B[1;33m             \u001B[1;32mraise\u001B[0m \u001B[0mTypeError\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'subclass {} should implement process method'\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mformat\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mname\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m      6\u001B[0m         \u001B[1;32mreturn\u001B[0m \u001B[0msuper\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0m__new__\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mcls\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mname\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mbases\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;31mTypeError\u001B[0m: subclass FaultPlugin should implement process method"
     ]
    }
   ],
   "source": [
    "class FaultPlugin(BasePlugin):\n",
    "\n",
    "    def run(self):\n",
    "        print('xxxx')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prerequisite-stream",
   "metadata": {
    "hidden": true
   },
   "source": [
    "1. FaultPlugin 继承于 BasePlugin ，从中获得元类 PluginMeta ；\n",
    "2. 当 Python 执行到 FaultPlugin 定义时，调用 PluginMeta 创建 FaultPlugin 类对象；\n",
    "3. 这时 PluginMeta.__new__ 开始执行，它检查类属性空间中是否包含 process 函数；\n",
    "4. 由于 FaultPlugin 未实现 process 函数，Python 将抛出 TypeError 异常，正如代码中写的那样；\n",
    "5. 正常情况下，PluginMeta.__new__ 将调用基类版本，即 type.__new__ 完成插件子类的创建；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eleven-myrtle",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在，我们把所有研究成果整合起来，形成一个完整的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "finite-repository",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T04:38:29.881706Z",
     "start_time": "2021-03-30T04:38:29.868724Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "subclass FaultPlugin should implement process method",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mTypeError\u001B[0m                                 Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-9-61b3a51c5e98>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m     32\u001B[0m \u001B[0mfoo\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mFooPlugin\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0minterval\u001B[0m\u001B[1;33m=\u001B[0m\u001B[1;36m5\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m     33\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m---> 34\u001B[1;33m \u001B[1;32mclass\u001B[0m \u001B[0mFaultPlugin\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mBasePlugin\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m     35\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m     36\u001B[0m     \u001B[1;32mdef\u001B[0m \u001B[0mrun\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mself\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;32m<ipython-input-9-61b3a51c5e98>\u001B[0m in \u001B[0;36m__new__\u001B[1;34m(cls, name, bases, dct)\u001B[0m\n\u001B[0;32m      5\u001B[0m     \u001B[1;32mdef\u001B[0m \u001B[0m__new__\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mcls\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mname\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mbases\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      6\u001B[0m         \u001B[1;32mif\u001B[0m \u001B[0mname\u001B[0m \u001B[1;33m!=\u001B[0m \u001B[1;34m'BasePlugin'\u001B[0m \u001B[1;32mand\u001B[0m \u001B[1;34m'process'\u001B[0m \u001B[1;32mnot\u001B[0m \u001B[1;32min\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 7\u001B[1;33m             \u001B[1;32mraise\u001B[0m \u001B[0mTypeError\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'subclass {} should implement process method'\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mformat\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mname\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m      8\u001B[0m         \u001B[1;32mreturn\u001B[0m \u001B[0msuper\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0m__new__\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mcls\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mname\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mbases\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mdct\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      9\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;31mTypeError\u001B[0m: subclass FaultPlugin should implement process method"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "class PlulginMeta(type):\n",
    "\n",
    "    def __new__(cls, name, bases, dct):\n",
    "        if name != 'BasePlugin' and 'process' not in dct:\n",
    "            raise TypeError('subclass {} should implement process method'.format(name))\n",
    "        return super().__new__(cls, name, bases, dct)\n",
    "\n",
    "class BasePlugin(metaclass=PlulginMeta):\n",
    "\n",
    "    def __init__(self, interval):\n",
    "        self.interval = interval\n",
    "\n",
    "    def serve_forever(self):\n",
    "        while True:\n",
    "            self.process()\n",
    "            time.sleep(self.interval)\n",
    "\n",
    "class BarPlugin(BasePlugin):\n",
    "\n",
    "    def process(self):\n",
    "        print('bar processing')\n",
    "\n",
    "bar = BarPlugin(interval=5)\n",
    "\n",
    "class FooPlugin(BasePlugin):\n",
    "\n",
    "    def process(self):\n",
    "        print('foo processing')\n",
    "\n",
    "foo = FooPlugin(interval=5)\n",
    "\n",
    "class FaultPlugin(BasePlugin):\n",
    "\n",
    "    def run(self):\n",
    "        print('xxxx')\n",
    "\n",
    "fault = FaultPlugin(interval=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "contrary-bahamas",
   "metadata": {
    "hidden": true
   },
   "source": [
    "例子所涉及了 类 对象、实例 对象以及 元类 对象，不同对象间关系可归纳如下：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_22.jpg\" width=800 height=600>  \n",
    "\n",
    "- 插件基类 BasePlugin ，插件子类 BarPlugin 、 FooPlugin 均为元类 PluginMeta 的实例；\n",
    "- 元类 PluginMeta 是 type 的子类，同时也是 type 的实例；\n",
    "- 由于执行 FaultPlugin 定义时抛了异常， FaultPlugin 类实例胎死腹中，它的实例 fault 更无从谈起；\n",
    "- 建议关系图与 **对象模型** 部分的对照阅读，进一步体会元类 PluginMeta 的位置、角色与作用；\n",
    "\n",
    "例子出于演示目的而设计，可能无法直接应用于工程项目，但相信它对启发元类应用思维会有所帮助。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verbal-testing",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## 奇门武动：如何实现代码热更新"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uniform-assistant",
   "metadata": {
    "hidden": true
   },
   "source": [
    "经过 Python 虚拟机、函数机制和类机制的学习，我们对 Python 程序执行过程的动态性已经了如指掌：\n",
    "\n",
    "- 在运行时，Python 可以动态创建 **函数** 对象；\n",
    "- 在运行时，Python 可以动态创建 **类** 对象；\n",
    "- 在运行时，Python 可以修改 **函数** 对象，改变它的行为；\n",
    "- 在运行时，Python 可以修改 **类** 对象，改变它的行为；\n",
    "- 在运行时，Python 可以动态编译代码并加入到虚拟机中执行；\n",
    "\n",
    "借助这些特性，我们可以实现程序运行时动态更新代码，也就是 代码热更新 ！\n",
    "\n",
    "对于一般程序而言，想要更新代码只有重启一条路。因此，拥有热更新能力的 Python 可以实现很不可思议的功能，具体如何进行呢？——我们从猴子补丁说起。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chief-column",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 猴子补丁"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "trained-bangladesh",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**猴子补丁** ( monkey patch )大家应该都听说过，这是一种在运行时添加、修改代码的技术，而无需修改源码。\n",
    "\n",
    "json 序列化是一个很常见的操作，在 Python 可以这样进行：\n",
    "\n",
    "    import json\n",
    "    json.dumps(some_data)\n",
    "    \n",
    "ujson 是另一个 json 序列化实现，由纯 C 语言编写，效率比标准库中的 json 模块更高，用法一样\n",
    "\n",
    "    import ujson\n",
    "    ujson.dumps(some_data)  \n",
    "    \n",
    "那么，如果想把整个程序中的 json 操作都换成 ujson ，该怎么办呢？\n",
    "\n",
    "直接引用 ujson 肯定是不行的，因为程序可能会引用第三方类库，我们肯定不想也不好改动第三方代码。以一个由 flask 框架实现的 api 为例，\n",
    "\n",
    "    from flask import Flask, jsonify\n",
    "\n",
    "    app = Flask(__name__)\n",
    "\n",
    "    @app.route('/')\n",
    "    def some_api():\n",
    "        return jsonify(some_data)\n",
    "        \n",
    "jsonify 函数用于响应 json 数据，它调用标准库 json 模块对数据进行 json 序列化，可 flask 并不是我们开发的。\n",
    "\n",
    "好在，利用 Python 执行过程的动态特性，我们可以在运行时替换 json 模块的相关函数实现。下面，我们编写 patch_json 函数，实现 dumps 和 loads 函数的替换：\n",
    "\n",
    "    import json\n",
    "    import ujson\n",
    "\n",
    "    def patch_json()\n",
    "        json.dumps = ujson.dumps\n",
    "        json.loads = ujson.loads\n",
    "\n",
    "    patch_json()\n",
    "    \n",
    "这样一来，只要 patch_json 函数成功执行，json 模块中的 dumps 、loads 函数就被换成了 ujson 版本。后续就算从 json 模块导入，最终得到的也是 ujson 版本！\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_23.jpg\" width=800 height=600>  \n",
    " \n",
    " 需要特别注意，json 模块属性在 patch_json 调用前就被直接引入，将不受 patch_json 控制：\n",
    "\n",
    "    import json\n",
    "    from json import dumps\n",
    "\n",
    "    patch_json()\n",
    "\n",
    "    # 执行 json 模块原来的版本，而不是 ujson 版本\n",
    "    dumps(some_data)\n",
    "    # 执行 ujson 版本\n",
    "    json.dumps(some_data)\n",
    "    \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_24.jpg\" width=800 height=600> \n",
    " \n",
    " 因此，许多应用猴子补丁的程序，在开头处便要执行替换逻辑，确保类似的现象不会发生。\n",
    "\n",
    "猴子补丁的应用范围很广，一般用来特换类库实现或者在单元测试中进行 mock 。诸如 greenlet 采用猴子补丁将阻塞的库函数替换成非阻塞的版本：\n",
    "\n",
    "    import gevent.monkey\n",
    "    gevent.monkey.patch_all()\n",
    "    \n",
    "由于猴子补丁可能会影响代码的可读性，应用不当可能导致一些奇怪的问题，因此不能滥用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooked-palace",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### reload"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sharing-costume",
   "metadata": {
    "hidden": true
   },
   "source": [
    "reload 函数相信大部分读者也有所耳闻了，它用于重新加载模块。与 Python 2 时代不同，reload 在最新的 Python 中不再作为内建函数存在了，而被移入标准库 importlib 模块中。\n",
    "\n",
    "假设我们有一个配置模块 config.py ，以变量形式定义着一些配置项：\n",
    "\n",
    "    wx = 'coding-fan'\n",
    "    title = '小菜学编程'\n",
    "    \n",
    "在程序中，我们只需将 config 模块导入，即可访问里面定义的每个配置项：\n",
    "\n",
    "    >>> import config\n",
    "    >>> print(config.title)\n",
    "    小菜学编程\n",
    "    \n",
    "现在，我们编辑 config.py 文件，将配置进行调整：\n",
    "\n",
    "    wx = 'fasionchan'\n",
    "    title = 'Python开发工程师'\n",
    "    \n",
    "如不做任何处理，程序无法获得调整后的配置，这一点都不意外：\n",
    "\n",
    "    >>> print(config.title)\n",
    "    小菜学编程\n",
    "    \n",
    "想要获取最新的配置，我们只能让 Python 重新加载 config 。操作也不复杂，调用 reload 函数即可：\n",
    "\n",
    "    >>> import importlib\n",
    "    >>> importlib.reload(config)\n",
    "    <module 'config' from '/Users/fasion/config.py'>\n",
    "    \n",
    "重新加载 config 模块后，我们成功获得最新配置，而程序完全无须重启！\n",
    "\n",
    "    >>> print(config.title)\n",
    "    Python开发工程师\n",
    "    \n",
    "借助 reload 函数，我们成功实现了一定程度的 代码热更新 能力！\n",
    "\n",
    "利用操作系统文件事件通知机制，我们还可以让 Python 在代码文件发生变化时自动加载新代码！\n",
    "\n",
    "有个坏消息，不同操作系统实现文件事件通知机制的方式完全不同，因此需要对主流操作系统进行适配。幸好，已经有先行者为此开发了 watchdog 包。我们只需用 pip 安装 watchdog ，即可站在巨人肩膀上开发。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cordless-resident",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import importlib\n",
    "import time\n",
    "import os.path\n",
    "\n",
    "from watchdog.observers import Observer\n",
    "from watchdog.events import FileSystemEventHandler, LoggingEventHandler\n",
    "\n",
    "import config\n",
    "\n",
    "class CodeEventHandler(FileSystemEventHandler):\n",
    "\n",
    "    def on_modified(self, event):\n",
    "        if event.src_path == config.__file__:\n",
    "            print('reloading config')\n",
    "            importlib.reload(config)\n",
    "\n",
    "def main():\n",
    "    observer = Observer()\n",
    "    observer.schedule(CodeEventHandler(), os.path.dirname(config.__file__), True)\n",
    "    observer.start()\n",
    "\n",
    "    while True:\n",
    "        print('title:', config.title)\n",
    "        time.sleep(5)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "shared-label",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这个例子先实现文件事件处理类 CodeEventHandler ，on_modified 方法接收修改事件。如果被修改文件刚好是 config 模块源码文件，我们调用 importlib.reload 重新加载 config 模块。\n",
    "\n",
    "接着在 main 函数，我们先初始化 watchdog 观测器 Observer ，然后启动它。这样一来，只要被观测路径上发生修改事件，watchdog 将调用 CodeEventHandler 对象的 on_modified 。\n",
    "\n",
    "注意到，我们不是直接观测 config.py 文件，而是递归观测其所在目录。如果直接观测目标文件，当它被删掉重建，便失去跟踪；相反，观测目标文件所在目录，重新创建的新文件也会得到跟踪。\n",
    "\n",
    "最后，程序进入主题逻辑，例子用一个周期性输出配置值 title 的循环来充当这个角色。\n",
    "\n",
    "把这个程序跑起来后，它将不断输出 config.title 的值；当 config.py 被修改后，它将输出 reloading config ，并重新加载 config 模块。\n",
    "\n",
    "现在，试着修改 config.py 中的 title 变量，程序将自动生效，无须重启：\n",
    "\n",
    "    title: Python开发工程师\n",
    "    reloading config\n",
    "    title: 小菜学编程\n",
    "    \n",
    "虽然例子中只涉及到一些简单的变量，但这种机制对诸如 函数 以及 类 等复杂对象也是支持的。接下来，我们将更进一步，充分认识 reload 机制的局限性，并探索破解局限性的方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "enormous-remove",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### reload 局限性"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conscious-overhead",
   "metadata": {
    "hidden": true
   },
   "source": [
    "通过 reload 函数重新加载模块，Python 将以原模块对象属性空间为全局/局部名字空间，再次执行模块代码。这种行为将导致一些诡异的现象，我们接着讨论。\n",
    "\n",
    "首先，旧模块变量不会被删除，除非在新模块代码中显式删除，这很好理解。\n",
    "\n",
    "假设模块 mo.py 原来有两个变量 a 和 b ：\n",
    "\n",
    "    a = 1\n",
    "    b = 2\n",
    "    \n",
    "模块导入后删掉 a 、修改 b 并新增 c ：\n",
    "\n",
    "    b = 22\n",
    "    c = 3\n",
    "    \n",
    "Python 重新加载模块 mo 时，以原模块对象属性空间为局部名字空间，执行新的模块代码。模块代码对 b 和 c 进行赋值，这样变引入的新变量 c ，变量 a 却被遗忘了，继续残留在模块中。关于模块加载以及模块代码执行过程，请参考虚拟机部分相关章节。\n",
    "\n",
    "不过，就算模块旧变量不删，最多也就是不够严谨而已，对于实现代码更新影响不大。\n",
    "\n",
    "另一个局限性影响就大了，reload 只会更新模块属性空间，对已暴露到外部的却无能为力。\n",
    "\n",
    "    >>> from mo import b\n",
    "    >>> print(b)\n",
    "    2\n",
    "    \n",
    "这段代码将 b 引入到当前局部名字空间，就不受模块 mo 约束了。当我们修改模块并重新加载后，b 保持不变：\n",
    "\n",
    "    >>> importlib.reload(mo)\n",
    "    >>> print(b)\n",
    "    2\n",
    "    \n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_25.jpg\" width=800 height=600> \n",
    " \n",
    " 如上图，我们可以清楚地看到，经过 reload 后，模块中的 b 得到了更新，但当前局部名字空间中的 b 保持不变。如果通过 mo 模块间接访问变量 b ，更新是可见的：\n",
    " \n",
    "     >>> print(mo.b)\n",
    "    22\n",
    "    \n",
    "因此，想要通过 reload 实现热更新，最好通过模块对象引用模块属性，不要直接导入。\n",
    "\n",
    "最后一个问题，如果在模块 mo 中 import 其他模块，reload 模块 mo 时，其他模块不会 reload 。这是要么显式 reload 其他模块，还要特别注意顺序；要么将其他模块路径从 sys.modules 中剔除，之后 Python 将重新加载它。\n",
    "\n",
    "可以将 sys.modules 理解成模块对象运行时缓存，Python 导入一个模块后，将以模块路径为 key ，以模块对象为 value 保存在该字典中。当同一模块被二次 import 时，Python 直接从这取出已加载的模块对象，避免重复加载。\n",
    "\n",
    "因此，当我们将某个模块从 sys.modules 中剔除，Python 将创建新的模块对象并执行模块代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "educational-romantic",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## 类和继承相关面试题精讲"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "looking-distributor",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 提供了完整的面向对象编程能力，将面向对象编程思想带到实际项目，可极大提高开发效率。因此，面向对象编程也是 Python 面试中必问的重要话题。\n",
    "\n",
    "想要在 Python 项目中应用面向对象编程技术，除了掌握基本的理论概念外，还要理解 Python 对象模型、 类机制、继承与属性查找的关系、描述符以及元类等诸多知识。这些都是面试中经常考察的关键知识点。\n",
    "\n",
    "本节精选若干典型面试题，以此抛砖引玉。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "threaded-spain",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 如何理解面向对象编程中的方法重写( overriding )和重载( overloading )？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unnecessary-sunday",
   "metadata": {
    "hidden": true
   },
   "source": [
    "\n",
    "\n",
    "请结合 Python 或其他编程语言进行说明。\n",
    "\n",
    "**方法重写** ( overriding )是指在子类中重新实现已在父类中定义的方法。\n",
    "\n",
    "在面向对象编程语言中，子类可以继承父类中的方法，而无须重新编写相同的方法。但有时子类并不想原封不动地继承父类所有功能，想对父类中的某些方法进行修改，这就需要采用方法重写特性。方法重写不能发生在同一个类中，只能发生在子类中。\n",
    "\n",
    "这是一个最简单的方法重写实例：\n",
    "\n",
    "    class Dog:\n",
    "\n",
    "        def eat(self):\n",
    "            print('yummy!')\n",
    "\n",
    "        def yelp(self):\n",
    "            print('woof!')\n",
    "\n",
    "    class Sleuth(Dog):\n",
    "\n",
    "        def yelp(self):\n",
    "            print('WOOF! WOOF! WOOF!')\n",
    "            \n",
    "Dog 是一个普通狗类，实现了 eat 和 yelp 方法。猎犬 Sleuth 继承于 Dog 类，因此继承了父类的 eat 方法。注意到，我们对 yelp 方法进行 重写 ，以连续三个大写的 WOOF 突出猎犬铿锵有力的吠声。\n",
    "\n",
    "这样一来，Sleuth 类继承了 Dog 中的 eat 方法，但自己的 yelp 方法覆盖了 Dog 中的相关定义：\n",
    "\n",
    "    >>> sleuth = Sleuth()\n",
    "    >>> sleuth.eat()\n",
    "    yummy!\n",
    "    >>> sleuth.yelp()\n",
    "    WOOF! WOOF! WOOF!\n",
    "    \n",
    "**重载** ( overloading )既可发生在同一个类的方法之间，一般称作方法重载；亦可发生在普通函数间，一般称作函数重载。这个特性允许开发人员定义名字相同，但输入参数不同的类方法或者普通函数，即同名方法/函数的不同版本。\n",
    "\n",
    "当程序调用重载方法或函数时，编译器将根据 参数个数 及 参数类型 ，自动绑定正确的版本。由于方法/函数绑定时涉及类型检查，因此一般只有静态类型编程语言才支持重载特性。\n",
    "\n",
    "Python 是一种动态类型编程语言，不支持方法重载，我们举一个简单的 C++ 程序作为例子：\n",
    "\n",
    "    #include <iostream>\n",
    "\n",
    "    using namespace std;\n",
    "\n",
    "    void print(int i) {\n",
    "        cout << \" Here is int \" << i << endl;\n",
    "    }\n",
    "\n",
    "    void print(double  f) {\n",
    "        cout << \" Here is float \" << f << endl;\n",
    "    }\n",
    "\n",
    "    void print(char const *c) {\n",
    "        cout << \" Here is char* \" << c << endl;\n",
    "    }\n",
    "\n",
    "    int main() {\n",
    "        print(10);\n",
    "        print(10.10);\n",
    "        print(\"ten\");\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "程序定义了 print 函数，分为 3 个不同版本，分别以整型、双精度浮点以及常字符串为参数。main 函数中调用 print 函数时，编译器将根据参数类型，自动选择正确的 print 函数版本。以 print(10) 为例，由于参数 10 是一个整数，编译器可以据此推导出 void print(int i) 版本。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moderate-geometry",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "###  Python 支持多继承吗？试说明多继承场景下实例对象类属性查找顺序？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "streaming-texture",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 支持多继承，只需将基类按顺序逐一声明即可：\n",
    "\n",
    "    class Child(Base1, Base2):\n",
    "        pass\n",
    "        \n",
    "当子类实例对象查找某个类属性时，先在子类 Child 中查找，再按定义顺序到基类中逐个中查找。如果基类也继承于其他类的基类，Python 将沿着继承链逐级回溯，最终来到 object 。\n",
    "\n",
    "类属性查找顺序决定程序的行为，不可不察，特别是在复杂多继承场景下。实际上，Python 类属性查找顺序是一个特殊的拓扑排序。这个拓扑排序首先是深度优先的，其次需要确保多继承基类按照定义的顺序查找。\n",
    "\n",
    "接下来，我们构造一个多继承关系网，考察决定属性查找顺序的重要因素，例子如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "alike-moderator",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:02:53.788542Z",
     "start_time": "2021-03-30T06:02:53.775535Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class A: pass\n",
    "\n",
    "class B: pass\n",
    "\n",
    "class C(A): pass\n",
    "\n",
    "class D(C): pass\n",
    "\n",
    "class E(B, A): pass\n",
    "\n",
    "class F(D, E): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "severe-anatomy",
   "metadata": {
    "hidden": true
   },
   "source": [
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_26.jpg\" width=800 height=600> \n",
    "子类总比父类先被搜索，因此必须满足以下关系，拓扑排序即可胜任：\n",
    "\n",
    "- F 先于 D ；\n",
    "- F 先于 E ；\n",
    "- D 先于 C ；\n",
    "- C 先于 A ；\n",
    "- A 先于 object ；\n",
    "- E 先于 A ；\n",
    "- E 先于 B ；\n",
    "- B 先于 object；\n",
    "\n",
    "而根据多继承基类列表顺序，必须保证：\n",
    "\n",
    "- B 先于 A ；\n",
    "- D 先于 E ；\n",
    "\n",
    "因此，F 最先被搜索，接着是 D ，然后按照深度优先的原则来到 C ；由于 B 先于 A ，不能接着搜索 A ；这时只能先搜索 E 分支，然后是 B ，再到 A ；A 和 B 皆搜索过后才能搜索 object 。因此，完整的搜索顺序是这样的：\n",
    "\n",
    "    F -> D -> C -> E -> B -> A -> object\n",
    "    \n",
    "Python 完成类对象初始化后，通过 C3 算法计算类属性搜索顺序，并将其保存在 __mro__ 属性中。我们可以据此确认推理结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "white-fetish",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:03:17.610675Z",
     "start_time": "2021-03-30T06:03:17.590701Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(__main__.F,\n",
       " __main__.D,\n",
       " __main__.C,\n",
       " __main__.E,\n",
       " __main__.B,\n",
       " __main__.A,\n",
       " object)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> F.__mro__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subject-messaging",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果规则前后矛盾，Python 将抛 TypeError 异常。这是一个典型的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "flexible-irrigation",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:05:09.737242Z",
     "start_time": "2021-03-30T06:05:09.728247Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Cannot create a consistent method resolution\norder (MRO) for bases A, B",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mTypeError\u001B[0m                                 Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-12-c231c21ecca3>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m      7\u001B[0m \u001B[1;32mclass\u001B[0m \u001B[0mD\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mB\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mA\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m \u001B[1;32mpass\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      8\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 9\u001B[1;33m \u001B[1;32mclass\u001B[0m \u001B[0mF\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mC\u001B[0m\u001B[1;33m,\u001B[0m \u001B[0mD\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m \u001B[1;32mpass\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mTypeError\u001B[0m: Cannot create a consistent method resolution\norder (MRO) for bases A, B"
     ]
    }
   ],
   "source": [
    "class A: pass\n",
    "\n",
    "class B: pass\n",
    "\n",
    "class C(A, B): pass\n",
    "\n",
    "class D(B, A): pass\n",
    "\n",
    "class F(C, D): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "downtown-shell",
   "metadata": {
    "hidden": true
   },
   "source": [
    "对于 F 类，基类 C 要求 A 先于 B 被搜索，而基类 D 要求 B 先于 A 被搜索，前后矛盾：\n",
    "\n",
    "由于多继承存在一定的歧义性，实际项目开发一般不鼓励复杂的多继承关系。如果多继承不可避免，则需要严谨确认类属性搜索顺序。最好查看 __mro__ 属性确认顺序符合预期，切勿想当然。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "altered-newfoundland",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 试设计装饰器 mystaticmethod ，实现与 staticmethod 相同的功能"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "preliminary-malpractice",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据属性描述符原理，我们需要实现一个装饰器，将函数对象改造成一个特殊的 **非数据描述符** 。当实例对象访问对应属性时，描述符 __get__ 方法将被调用，它只需将函数对象原样返回即可：\n",
    "\n",
    "    class mystaticmethod:\n",
    "\n",
    "        def __init__(self, func):\n",
    "            self.func = func\n",
    "\n",
    "        def __get__(self, instance, owner):\n",
    "            return self.func\n",
    "            \n",
    "这样一来，静态函数可以这样来写，以在 Foo 类实现静态方法 add 为例：\n",
    "\n",
    "    class Foo:\n",
    "\n",
    "        @mystaticmethod\n",
    "        def add(a, b):\n",
    "            return a + b\n",
    "\n",
    "        def add2(a, b):\n",
    "            return a + b\n",
    "            \n",
    "接着，我们创建一个实例对象 foo ，并通过实例对象访问 add 属性。我们得到的是原始的 add 函数对象，而不是一个 bound method 对象：\n",
    "\n",
    "    >>> foo = Foo()\n",
    "    >>> foo.add\n",
    "    <function Foo.add at 0x1092189d8>\n",
    "    >>> foo.add(1, 2)\n",
    "    3\n",
    "\n",
    "至此，我们通过自己的聪明才智，成功实现了静态方法装饰器。作为对照，add2 未加装饰，它将成为一个普通的类方法。我们通过 foo 实例对象访问 add2 属性时，将得到一个 bound method 对象：\n",
    "\n",
    "    >>> foo.add2\n",
    "    <bound method Foo.add2 of <__main__.Foo object at 0x1092349e8>>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaged-ceremony",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Python 如何实现单例模式？试举例说明。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loose-tuesday",
   "metadata": {
    "hidden": true
   },
   "source": [
    "通过 Python 对象模型部分学习，我们知道 Python 对象创建和初始化分别由定义于类型对象中的 tp_new 以及 tp_init 函数负责。相应地，在自定义类中，可以通过 __new__ 和 __init__ 魔术方法来控制实例对象的实例化。\n",
    "\n",
    "对于类 X ，当我们调用 x = X() 创建实例对象 x 时，Python 内部分为两步进行：\n",
    "\n",
    "1. 调用 X.__new__ 为实例对象分配内存，这一步完成实例对象的创建；\n",
    "2. 调用 X.__init__ 将实例对象初始化，1 中生成的实例对象作为 self 参数传给 __init__ 方法；\n",
    "如果 X 类未实现 __new__ 方法，Python 将使用其父类的。如果父类也没有实现该方法，Python 最终将调用 object.tp_new 。object 基类型对象是所有类型对象的基类，它提供了一个通用的 tp_new 版本。\n",
    "\n",
    "当 __init__ 方法执行时，对象已经完成了创建，因此无法实现全局唯一的约束。但我们可以在 __new__ 方法中做手脚，先判断对象是否已经创建，如果是直接将其返回，否则调用父类的 __new__ 方法完成进行创建并保存。示例代码如下：\n",
    "\n",
    "    class SomeClass:\n",
    "\n",
    "        instance = None\n",
    "\n",
    "        def __new__(cls):\n",
    "            if cls.instance is None:\n",
    "                cls.instance = super().__new__(cls)\n",
    "\n",
    "            return cls.instance\n",
    "\n",
    "        def __init__(self):\n",
    "            # do some initialization\n",
    "            pass\n",
    "            \n",
    "例子中，类属性 instance 用于保存 SomeClass 类全局唯一的实例对象。\n",
    "\n",
    "在 __new__ 函数中，我们先检查 instance 属性。如果它为 None ，说明实例尚未创建。这时，我们通过 super 调用父类的 __new__ 方法完成实例对象的创建，并将其保存与 instance 属性中。\n",
    "\n",
    "当我们再次调用 SomeClass 类创建实例对象时，__new__ 方法也将被调用。但此时，instance 已不再是 None ， __new__ 直接将其返回，由此避免重复实例化。\n",
    "\n",
    "这样一来，不管我们调用 SomeClass 多少次，得到的实例对象总是全局唯一的那一个：\n",
    "\n",
    "    >>> instance = SomeClass()\n",
    "    >>> id(instance)\n",
    "    4302595728\n",
    "    >>> instance = SomeClass()\n",
    "    >>> id(instance)\n",
    "    4302595728"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "historical-mount",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 如果程序中有成千上万的 User 类实例对象，如何优化内存使用？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suspected-interstate",
   "metadata": {
    "hidden": true
   },
   "source": [
    "提示 User 只包含属性 name 和 email 以及若干个类方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "novel-compact",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:11:36.435272Z",
     "start_time": "2021-03-30T06:11:36.425283Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class User:\n",
    "    \n",
    "    def __init__(self, name, email):\n",
    "        self.name = name\n",
    "        self.email = email\n",
    "        \n",
    "    def some_method(self):\n",
    "        pass\n",
    "        \n",
    "    def another_method(self):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "assured-behalf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:11:41.931603Z",
     "start_time": "2021-03-30T06:11:41.924594Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<__main__.User at 0x2a6df22fa90>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user = User(name='小菜学编程', email='coding-fan@mp.weixin.qq.com')\n",
    "user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sized-liability",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据我们学到的 Python 类机制知识，User 类对象及其实例对象底层内存布局大致如下：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_27.jpg\" width=800 height=600> \n",
    " \n",
    " 中间橘色部分是 User 类对象，它是一个 PyTypeObject 结构体，tp_dict 字段指向类属性空间。类属性空间是一个字典对象，里面保存着类属性，类方法 some_method 也位于其中。\n",
    "\n",
    "而 User 实例对象(如例子中的 user )，如蓝色部分所示，它的结构则简单得多，底层是一个 PyObject 结构体。紧接着 PyObject 的是一个指针，指向实例对象属性空间的字典。实例对象属性空间也是一个字典对象，里面则保存着实例对象的属性 name 和 email 。\n",
    "\n",
    "注意到，User 类对象全局只有一个，而它的实例对象却可以有无数个。那么，一个实例对象占用多少内存空间呢？我们调用 sys.getsizeof 查看一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fixed-indonesia",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:13:30.477735Z",
     "start_time": "2021-03-30T06:13:30.467756Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.getsizeof(user)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "covered-peninsula",
   "metadata": {
    "hidden": true
   },
   "source": [
    "不对吧？在内建对象部分，我们学习了字典对象的布局，它动辄 200 多字节呢！实际上，getsizeof 只计算蓝色部分内存，这部分是 32 字节，而 PyObject 之前还有 32 字节用于垃圾回收的隐形开销。\n",
    "\n",
    "因此，一个 User 实例对象的内存开销是 64 字节加上属性空间字典对象的内存开销。换句话讲，每创建一个 User 实例，Python 都会帮你创建一个字典对象，来保存实例属性。\n",
    "\n",
    "属性空间字典是一个不小的开销，当前这个场景似乎不需要。因为 User 实例对象属性只有固定的两个，不会出现这样的情况：\n",
    "\n",
    "    >>> user.age = 10\n",
    "    >>> user.age\n",
    "    10\n",
    "    \n",
    "联想到函数局部名字空间与这个场景非常类似：参数个数是固定的，因此名字空间由一个静态的数组实现，而不是用字典对象。那么，实例属性是不是也可以采用类似的思路呢？答案是肯定的，Python 实现了 slot 机制："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "composite-preserve",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:15:07.123707Z",
     "start_time": "2021-03-30T06:15:07.116696Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class User:\n",
    "    \n",
    "    __slots__ = ('name', 'email')\n",
    "    \n",
    "    def __init__(self, name, email):\n",
    "        self.name = name\n",
    "        self.email = email\n",
    "        \n",
    "    def some_method(self):\n",
    "        pass\n",
    "        \n",
    "    def another_method(self):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "identical-cinema",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:15:08.233719Z",
     "start_time": "2021-03-30T06:15:08.226720Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "user = User(name='小菜学编程', email='coding-fan@mp.weixin.qq.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "supreme-hamburg",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这个新的 User 类通过 __slots__ 将属性 name 和 email 申明为以 slot 形式实现。采用 slot 机制后，Python 便不会为 User 实例对象分配属性空间字典了，而是将属性作为 slot 依次排列在 PyObject 结构体之后："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "developmental-bumper",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:17:56.974861Z",
     "start_time": "2021-03-30T06:17:56.956884Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.getsizeof(user)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "armed-egypt",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:16:47.829159Z",
     "start_time": "2021-03-30T06:16:47.817151Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'User' object has no attribute '__dict__'",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mAttributeError\u001B[0m                            Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-21-8ef458f7e78f>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0muser\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0m__dict__\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mAttributeError\u001B[0m: 'User' object has no attribute '__dict__'"
     ]
    }
   ],
   "source": [
    "user.__dict__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "living-domestic",
   "metadata": {
    "hidden": true
   },
   "source": [
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_28.jpg\" width=800 height=600> \n",
    "\n",
    "由此一来，动辄几百字节的字典对象被节省下来，内存使用效率得到有效提升。当然了，天下没有免费的午餐，我们丧失了在运行时新增属性的灵活性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "qualified-crime",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:17:42.656503Z",
     "start_time": "2021-03-30T06:17:42.640512Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'User' object has no attribute 'age'",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mAttributeError\u001B[0m                            Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-22-4a07730afef0>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0muser\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mage\u001B[0m \u001B[1;33m=\u001B[0m \u001B[1;36m10\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mAttributeError\u001B[0m: 'User' object has no attribute 'age'"
     ]
    }
   ],
   "source": [
    "user.age = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "offshore-concert",
   "metadata": {
    "hidden": true
   },
   "source": [
    "因此，slot 机制只适用于属性个数固定的场景。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "turkish-jason",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 用 type 元类型创建一个等价的 MyFloat 类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "enhanced-print",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class MyFloat(float):\n",
    "    \n",
    "    def round(self):\n",
    "        return int(self)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "studied-rwanda",
   "metadata": {
    "hidden": true
   },
   "source": [
    "根据 Python 对象模型，type 是所有类型对象的类型，调用 type 可以创建新的类型对象。在类机制部分，我们学习了自定义类的创建过程，知道 type 在其中所起的重要作用。\n",
    "\n",
    "使用 type 创建新类型对象，接口如下：\n",
    "\n",
    "    type(name, bases, dict) -> a new type\n",
    "    \n",
    "- name ，新类型名；\n",
    "- bases ，基类列表；\n",
    "- dict ，类属性空间；\n",
    "\n",
    "显而易见，为创建与 MyFloat 完全等价的类对象，name 必须是 MyFloat ；基类是 float 对象，因而 bases 必须是由 float 组成的元组，即 (float,) ；而作为类属性空间 dict 必须包含 round 函数对象。\n",
    "\n",
    "我们先定义 round 函数以及类属性空间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "secret-brisbane",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:19:24.124334Z",
     "start_time": "2021-03-30T06:19:24.112327Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> def round(self):\n",
    "...     return int(self)\n",
    "...\n",
    ">>> attrs = {'round': round}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "checked-alaska",
   "metadata": {
    "hidden": true
   },
   "source": [
    "接着，我们调用 type 元类型对象完成新类对象的创建："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "imperial-stanford",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:19:38.680060Z",
     "start_time": "2021-03-30T06:19:38.661070Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "__main__.MyFloat"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> MyFloat = type('MyFloat', (float,), attrs)\n",
    ">>> MyFloat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "challenging-banana",
   "metadata": {
    "hidden": true
   },
   "source": [
    "最后，我们验证新 MyFloat 类的功能，确认它与例子程序完全等价："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "divine-march",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:19:55.057632Z",
     "start_time": "2021-03-30T06:19:55.039645Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> pi = MyFloat('3.14')\n",
    ">>> pi.round()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "postal-perth",
   "metadata": {
    "hidden": true
   },
   "source": [
    "type 元类编程提供了在运行时动态创建新类型的强大能力，在框架设计中非常有用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "middle-delight",
   "metadata": {},
   "source": [
    "# 生成器与协程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prescription-usage",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 重新认识生成器 generator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "significant-valentine",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 基本用法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reported-relay",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们知道，函数体包含 yield 关键字的函数不是一个普通函数。这种函数叫做 生成器 ( generator )，一般用于循环处理结构，应用得当可以极大优化内存使用效率。例如，设计一个函数，打开文件并将每一行转成大写并返回：\n",
    "\n",
    "    def read_file_upper(path):\n",
    "        lines = []\n",
    "        with open(path) as f:\n",
    "            for line in f:\n",
    "                lines.append(line.upper())\n",
    "        return lines\n",
    "    \n",
    "这个版本的函数，在内部创建了一个 list 对象，用于保存转换结果。for 循环则遍历文件每一行，将其转成大写并追加到列表中。这样一来，文件中的每一行均需要保存在列表中，如果文件很大，内存开销可想而知。\n",
    "\n",
    "我们可以借助 yield 关键字，将 read_file_upper 函数改成生成器版本。函数主体逻辑没有任何变化，只是将每行数据的处理结果通过 yield 逐个返回，而不是收集到 list 对象后再返还。\n",
    "\n",
    "    def iter_file_upper(path):\n",
    "        with open(path) as f:\n",
    "            for line in f:\n",
    "                yield line.upper()\n",
    "                \n",
    "如果现在有一个文本文件 data.txt ，里面包含以下内容：\n",
    "\n",
    "    hello, world\n",
    "    life is short, use python\n",
    "    my wechat id is: coding-fan\n",
    "    bye\n",
    "    \n",
    "用 iter_file_upper 生成器，我们可以这样对它进行处理：\n",
    "\n",
    "    >>> for line in iter_file_upper('text.txt'):\n",
    "    ...     print(line.strip())\n",
    "    HELLO, WORLD\n",
    "    LIFE IS SHORT, USE PYTHON\n",
    "    MY WECHAT ID IS: CODING-FAN\n",
    "    BYE\n",
    "    \n",
    "iter_file_upper 生成器用法与 read_file_upper 函数大致相同，但它不会一次性拿住文件所有数据行，而是逐行处理、逐个返回，这样便将内存使用量降到最低。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "grave-exercise",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 行为观察"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "forbidden-actress",
   "metadata": {
    "hidden": true
   },
   "source": [
    "那么，生成器为什么会有这样的奇效呢？我们接着观察：\n",
    "\n",
    "    >>> g = iter_file_upper('text.txt')\n",
    "    >>> g\n",
    "    <generator object iter_file_upper at 0x103becd68>\n",
    "    \n",
    "我们调用 iter_file_upper 后，得到一个生成器对象，而不是文件处理结果，这时 iter_file_upper 还未开始执行。\n",
    "\n",
    "当我们调用 next 函数从生成器接收下一个数据时，iter_file_upper 开始执行并在 yield 处停下来，并把第一行的处理结果返回给我们：\n",
    "\n",
    "    >>> next(g)\n",
    "    'HELLO, WORLD\\n'\n",
    "    \n",
    "这时，生成器处于暂停状态，没有我们的指令，它不会接着处理第二行数据。\n",
    "\n",
    "当我们再次执行 next 函数时，生成器再次恢复执行，处理下一行数据并在 yield 处再次暂停：\n",
    "\n",
    "    >>> next(g)\n",
    "    'LIFE IS SHORT, USE PYTHON\\n'    \n",
    "\n",
    "生成器记住了自己的执行进度，每次调用 next 函数，它总是处理并生产下一个数据，完全不用我们瞎操心：\n",
    "\n",
    "    >>> next(g)\n",
    "    'MY WECHAT ID IS: CODING-FAN\\n'\n",
    "    >>> next(g)\n",
    "    'BYE\\n'\n",
    "    \n",
    "当 iter_file_upper 代码逻辑执行完毕，它将给 next 抛一个异常，以此通知调用者它已经结束了：\n",
    "\n",
    "    >>> next(g)\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "    StopIteration\n",
    "    \n",
    "因此，我们可以简单认为 for-in 循环在 Python 虚拟机内部是这样实现的：\n",
    "\n",
    "- 不断调用 next 函数让生成器产出数据；\n",
    "- 直到生成器抛出 StopIteration 异常；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "retired-absorption",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 任务上下文"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nominated-davis",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在经典的线程模型中，每个线程有一个独立的执行流，只能执行一个任务。如果一个程序需要同时处理多个任务，可以借助 **多进程** 或者 **多线程** 技术。假设一个站点需要同时服务多个客户端连接，可以为每个连接创建一个独立的线程进行处理。\n",
    "\n",
    "不管线程还是进程，切换时都会带来巨大的开销：用户态/内核态切换、执行上下文保存和恢复、CPU缓存刷新等等。因此，用线程或进程来驱动小任务的执行，显然不是一个理想的选择。\n",
    "\n",
    "那么，除了线程和进程，还有其他解决方案吗？开始讨论之前，我们先来总结实现多任务执行系统的关键之处。\n",
    "\n",
    "一个程序想要同时处理多个任务，必须提供一种能够记录任务执行进度的机制。在经典线程模型中，这个机制由 CPU 提供：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_29.jpg\" width=800 height=600> \n",
    "\n",
    "如上图，程序内存空间分为代码、数据、堆以及栈等多个段，CPU 中的 CS 寄存器指向代码段，SS 寄存器指向栈段。当程序任务(线程)执行时，IP 寄存器指向代码段中当前正被执行的指令，BP 寄存器指向当前栈帧，SP 寄存器则指向栈顶。\n",
    "\n",
    "有了 IP 寄存器，CPU 可以取出需要执行的下一条指令；有了 BP 寄存器，当函数调用结束时，CPU 可以回到调用者继续执行。因此，CPU 寄存器与内存地址空间一起构成了任务执行上下文，记录着任务执行进度。当任务切换时，操作系统先将 CPU 当前寄存器保存到内存，然后恢复待执行任务的寄存器。\n",
    "\n",
    "至此，我们已经受到一些启发：生成器不是可以记住自己的执行进度吗？那么，是不是可以用生成器来实现任务执行流？由于生成器在用户态运行，切换成本比线程或进程小很多，是组织微型任务的理想手段。\n",
    "\n",
    "现在，我们用生成器来写一个玩具协程，以此体会协程的运行机制："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "whole-hearing",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:41:31.265024Z",
     "start_time": "2021-03-30T06:41:31.254018Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def co_process(data):\n",
    "    print('task with data {} started'.format(data))\n",
    "\n",
    "    yield\n",
    "    print('step one for data {} finished'.format(data))\n",
    "\n",
    "    yield\n",
    "    print('step two for data {} finished'.format(data))\n",
    "\n",
    "    yield\n",
    "    print('step there for data {} finished'.format(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respective-fishing",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程 co_process 用于处理数据 data ，处理分为 3 步，每一步间可能发生任务切换，即通过 yield 让出执行权。\n",
    "\n",
    "接着，我们创建 3 个协程，用于处理 3 个不同的数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "substantial-poetry",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:41:32.871345Z",
     "start_time": "2021-03-30T06:41:32.866362Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> t1 = co_process('a')\n",
    ">>> t2 = co_process('b')\n",
    ">>> t3 = co_process('c')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "careful-guitar",
   "metadata": {
    "hidden": true
   },
   "source": [
    "至此，协程已经完成初始化，但暂未开始执行。我们需要调用 next 函数，将他们逐个激活："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "dental-alloy",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:41:46.478599Z",
     "start_time": "2021-03-30T06:41:46.466607Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task with data a started\n",
      "task with data b started\n",
      "task with data c started\n"
     ]
    }
   ],
   "source": [
    ">>> next(t1)\n",
    ">>> next(t2)\n",
    ">>> next(t3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "second-angola",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程激活后开始执行，开始输出提示信息，直到遇到第一个 yield 语句。yield 将让出协程的执行权，执行权又回到我们手中。在实际项目中，协程一般在等待 IO 时，才会通过 yield 语句让出执行权。\n",
    "\n",
    "注意到，在这个例子中，我们扮演着调度器的角色，手中掌握着协程调度权——让第一个协程任务往下执行两步："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "authorized-economics",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:42:06.112129Z",
     "start_time": "2021-03-30T06:42:06.108144Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step one for data a finished\n",
      "step two for data a finished\n"
     ]
    }
   ],
   "source": [
    ">>> next(t1)\n",
    ">>> next(t1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "explicit-portfolio",
   "metadata": {
    "hidden": true
   },
   "source": [
    "再让第二三个协程交替执行两步："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "impaired-reunion",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:42:17.515290Z",
     "start_time": "2021-03-30T06:42:17.499287Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step one for data b finished\n",
      "step one for data c finished\n"
     ]
    }
   ],
   "source": [
    ">>> next(t2)\n",
    ">>> next(t3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "sensitive-hotel",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:42:21.232916Z",
     "start_time": "2021-03-30T06:42:21.221922Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step two for data b finished\n",
      "step two for data c finished\n"
     ]
    }
   ],
   "source": [
    ">>> next(t2)\n",
    ">>> next(t3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broken-designation",
   "metadata": {
    "hidden": true
   },
   "source": [
    "请注意，在实际项目中，一般由 **事件循环** 充当调度器的角色，而不是通过我们的双手。在《手把手教你设计协程库》一节，我们将一起探讨实现事件循环的思路。\n",
    "\n",
    "我们接着调度协程任务，当某个协程执行完毕，它将通过 next 函数抛出 StopIteration 异常，以此通知我们："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "identified-float",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:42:49.966429Z",
     "start_time": "2021-03-30T06:42:49.958422Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step there for data a finished\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-34-463776c2fe83>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mnext\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mt1\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m      2\u001B[0m \u001B[0mnext\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mt2\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m      3\u001B[0m \u001B[0mnext\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mt3\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;31mStopIteration\u001B[0m: "
     ]
    }
   ],
   "source": [
    ">>> next(t1)\n",
    ">>> next(t2)\n",
    ">>> next(t3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amended-accused",
   "metadata": {
    "hidden": true
   },
   "source": [
    "虽然这只是一个玩具程序，几乎没有任何实际用途。然而，正是这个玩具，帮我们解开实现协程的全部秘密！\n",
    "\n",
    "下一小节，我们将再接再厉，通过字节码进一步吃透协程的运行原理。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considerable-fundamental",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 字节码解读 generator 运行机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inappropriate-rehabilitation",
   "metadata": {
    "hidden": true
   },
   "source": [
    "上一小节，我们重新考察了 生成器 ( generator ) 的运行时行为，发现了它神秘的一面。生成器函数体代码可以通过 yield 关键字暂停和恢复执行，这个特性可以用来实现 **协程** 。\n",
    "\n",
    "那么，生成器诸多神奇特性是如何实现的呢？为了更好地理解生成器协程，我们有必要深入研究它的运行机制。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sporting-track",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 生成器的创建"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "living-wallet",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们对上节中的 co_process 生成器略加修改后作为研究对象，继续深入字节码，力求洞察生成器执行的原理："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "controlling-handle",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:35:43.687639Z",
     "start_time": "2021-03-30T07:35:43.673648Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def co_process(arg):\n",
    "    print('task with argument {} started'.format(arg))\n",
    "    \n",
    "    data = yield 1\n",
    "    print('step one finished， got {} from caller'.format(data))\n",
    "    \n",
    "    data = yield 2\n",
    "    print('step two finished， got {} from caller'.format(data))\n",
    "    \n",
    "    data = yield 3\n",
    "    print('step three finished， got {} from caller'.format(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinct-solid",
   "metadata": {
    "hidden": true
   },
   "source": [
    "co_process 是一个特殊的函数对象，它被调用后并不会立刻执行函数体，而是得到一个生成器对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "knowing-works",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:53:26.479529Z",
     "start_time": "2021-03-30T06:53:26.467536Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.co_process(arg)>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "co_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "serial-worse",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:53:36.980481Z",
     "start_time": "2021-03-30T06:53:36.974485Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object co_process at 0x000002A6DF91F890>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genco = co_process('foo')\n",
    "genco"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "incorrect-spelling",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:53:43.594458Z",
     "start_time": "2021-03-30T06:53:43.586462Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "generator"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genco.__class__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blank-promise",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在函数机制部分，我们知道函数调用由 CALL_FUNCTION 字节码负责：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "banned-turtle",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:54:00.344065Z",
     "start_time": "2021-03-30T06:54:00.332058Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1           0 LOAD_NAME                0 (co_process)\n",
      "              2 LOAD_CONST               0 ('foo')\n",
      "              4 CALL_FUNCTION            1\n",
      "              6 POP_TOP\n",
      "              8 LOAD_CONST               1 (None)\n",
      "             10 RETURN_VALUE\n"
     ]
    }
   ],
   "source": [
    ">>> import dis\n",
    ">>> dis.dis(compile(\"co_process('foo')\", '', 'exec'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "saving-lighter",
   "metadata": {
    "hidden": true
   },
   "source": [
    "那么，什么情况下函数调用会返回生成器呢？顺着 CALL_FUNCTION 字节码处理逻辑，不难找到答案。\n",
    "\n",
    "CALL_FUNCTION 字节码在 Python/ceval.c 中处理，它主要是调用 call_function 函数完成工作。call_function 函数根据被调用对象类型区别处理，可分为 **类方法** 、 **函数对象** ， **普通可调用对象** 等等。\n",
    "\n",
    "在这个例子中，被调用对象是函数对象。因此，call_function 函数调用位于 Objects/call.c 中的 _PyFunction_FastCallKeywords 函数，而它则进一步调用位于 Python/ceval.c 的 _PyEval_EvalCodeWithName 函数。\n",
    "\n",
    "_PyEval_EvalCodeWithName 函数先为目标函数 co_process 创建 **栈帧** 对象 f，然后检查代码对象标识。若代码对象带有 CO_GENERATOR 、CO_COROUTINE 或 CO_ASYNC_GENERATOR 标识，便创建生成器并返回：\n",
    "\n",
    "    /* Handle generator/coroutine/asynchronous generator */\n",
    "    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) {\n",
    "        PyObject *gen;\n",
    "        PyObject *coro_wrapper = tstate->coroutine_wrapper;\n",
    "        int is_coro = co->co_flags & CO_COROUTINE;\n",
    "        \n",
    "        // 省略\n",
    "\n",
    "        /* Create a new generator that owns the ready to run frame\n",
    "         * and return that as the value. */\n",
    "        if (is_coro) {\n",
    "            gen = PyCoro_New(f, name, qualname);\n",
    "        } else if (co->co_flags & CO_ASYNC_GENERATOR) {\n",
    "            gen = PyAsyncGen_New(f, name, qualname);\n",
    "        } else {\n",
    "            gen = PyGen_NewWithQualName(f, name, qualname);\n",
    "        }\n",
    "        if (gen == NULL) {\n",
    "            return NULL;\n",
    "        }\n",
    "        \n",
    "        // 省略\n",
    "\n",
    "        return gen;\n",
    "    }\n",
    "    \n",
    "代码对象标识 co_flags 在编译时由语法规则确定，通过 co_process ，我们可以找到其代码对象标识："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "french-alaska",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T06:55:20.184816Z",
     "start_time": "2021-03-30T06:55:20.169824Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> co_process.__code__.co_flags"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laden-engineering",
   "metadata": {
    "hidden": true
   },
   "source": [
    "CO_GENERATOR 宏定义于 Include/code.h 头文件，它的值是 0x20 ，co_process 代码对象确实带有该标识："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "minor-charlotte",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:20:24.019949Z",
     "start_time": "2021-03-30T07:20:24.000960Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> co_process.__code__.co_flags  & 0x20"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "after-platform",
   "metadata": {
    "hidden": true
   },
   "source": [
    "注意到，用于保存 co_process 函数执行上下文的栈帧对象 f ，作为一个重要字段保存于生成器对象 gen 中：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_30.jpg\" width=800 height=600> \n",
    " \n",
    " 至此，生成器对象的创建过程已经浮出水面。与普通函数一样，当 co_process 被调用时，Python 将为其创建栈帧对象，用于维护函数执行上下文 —— **代码对象** 、 **全局名字空间** 、 **局部名字空间** 以及 **运行栈** 都在其中。\n",
    "\n",
    "与普通函数不同的是，co_process 代码对象带有生成器标识。Python 不会立即执行代码对象，栈帧对象也不会被接入调用链，因此 f_back 字段是空的。相反，Python 创建了一个生成器对象，并将其作为函数调用结果返回。\n",
    "\n",
    "生成器对象底层由 PyGenObject 结构体表示，定义于 Include/genobject.h 头文件中。生成器类型对象同样由 PyTypeObject 结构体表示，全局只有一个，以全局变量的形式定义于 Objects/genobject.c 中，也就是 PyGen_Type 。\n",
    "\n",
    "PyGenObject 结构体中的字段也很好理解，顾名即可思义，这也体现了变量名的作用：\n",
    "\n",
    "- ob_refcnt ，引用计数 ，这是任何对象都包含的公共字段；\n",
    "- ob_type ，对象类型 ，指向其类型对象，这也是任何对象都包含的公共字段；\n",
    "- gi_frame ，生成器执行时所需的 栈帧对象 ，用于保存执行上下文信息；\n",
    "- gi_running ，标识生成器是否运行中；\n",
    "- gi_code ，代码对象 ；\n",
    "- gi_weakreflist ，弱引用相关，不深入讨论；\n",
    "- gi_name ，生成器名；\n",
    "- gi_qualname ，同上；\n",
    "- gi_exec_state ，生成器执行状态；\n",
    "\n",
    "最后，可以在 Python 中访问生成器对象 genco ，进一步印证我们在源码中得到的结论"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "protecting-congo",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:00:59.948046Z",
     "start_time": "2021-03-30T07:00:59.929043Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 生成器创建后，尚未开始执行\n",
    ">>> genco.gi_running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "pacific-reynolds",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:01:01.875016Z",
     "start_time": "2021-03-30T07:01:01.869032Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<frame at 0x000002A6DDAD14E0, file '<ipython-input-35-ed5a08f5dc3e>', line 1, code co_process>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 栈帧对象\n",
    ">>> genco.gi_frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "anticipated-oxford",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:01:19.714725Z",
     "start_time": "2021-03-30T07:01:19.709726Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<code object co_process at 0x000002A6DF2DE500, file \"<ipython-input-35-ed5a08f5dc3e>\", line 1>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 生成器和栈帧的代码对象，均来自 co_process 函数对象\n",
    ">>> genco.gi_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "modern-plant",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:01:21.011407Z",
     "start_time": "2021-03-30T07:01:21.001415Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<code object co_process at 0x000002A6DF2DE500, file \"<ipython-input-35-ed5a08f5dc3e>\", line 1>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> genco.gi_frame.f_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "french-newcastle",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:01:30.178772Z",
     "start_time": "2021-03-30T07:01:30.171777Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<code object co_process at 0x000002A6DF2DE500, file \"<ipython-input-35-ed5a08f5dc3e>\", line 1>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> co_process.__code__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intermediate-worthy",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 生成器的执行"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fewer-palestinian",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在前面例子中，co_process 函数被调用后，返回生成器 genco 。这时，函数体尚未开始执行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "general-douglas",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:02:08.529812Z",
     "start_time": "2021-03-30T07:02:08.521818Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genco.gi_frame.f_lasti"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "official-vegetarian",
   "metadata": {
    "hidden": true
   },
   "source": [
    "栈帧对象 f_lasti 字段记录当前字节码执行进度，-1 表示尚未开始执行。\n",
    "\n",
    "经过前一小节学习，我们知道：借助 next 内建函数或者 send 方法可以启动生成器，并驱动它不断执行。这意味着，生成器执行的秘密可以通过这两个函数找到。\n",
    "\n",
    "我们先从 next 函数入手，作为内建函数，它定义于 Python/bltinmodule.c 源文件，C 语言函数 builtin_next 是也。builtin_next 函数逻辑非常简单，除了类型检查等样板式代码，最关键的是这一行：\n",
    "\n",
    "    res = (*it->ob_type->tp_iternext)(it);\n",
    "    \n",
    "这行代码表明，next 函数实际上调用了生成器类型对象的 tp_iternext 函数完成工作。这听上去有些拗口，用 Python 的语言来描述就清晰多了 —— next(genco) 等价于："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "academic-orlando",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:03:20.644320Z",
     "start_time": "2021-03-30T07:03:20.627344Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task with argument foo started\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> genco.__class__.__next__(genco)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "meaningful-fabric",
   "metadata": {
    "hidden": true
   },
   "source": [
    "还记得 Python **对象模型** 部分内容吗？类型对象决定实例对象的行为，实例对象相关操作函数的指针都保存在类型对象中。生成器作为 Python 对象中的一员，当然也遵守这一法则。\n",
    "\n",
    "顺着生成器类型对象的肉身 PyGen_Type ( Objects/genobject.c )，很快就可以摸到 gen_iternext 函数。\n",
    "\n",
    "另一方面， genco.send 也可以启动并驱动生成器的执行，根据 Objects/genobject.c 中的方法定义，它底层调用 _PyGen_Send 函数：\n",
    "\n",
    "    static PyMethodDef gen_methods[] = {\n",
    "        {\"send\",(PyCFunction)_PyGen_Send, METH_O, send_doc},\n",
    "        {\"throw\",(PyCFunction)gen_throw, METH_VARARGS, throw_doc},\n",
    "        {\"close\",(PyCFunction)gen_close, METH_NOARGS, close_doc},\n",
    "        {NULL, NULL}        /* Sentinel */\n",
    "    };\n",
    "    \n",
    "不管 gen_iternext 函数还是 _PyGen_Send 函数，都是直接调用 gen_send_ex 函数完成工作的：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_31.jpg\" width=800 height=600> \n",
    "\n",
    "因此，不管是执行 next(genco) 还是 genco.send(None) ，最终都由 gen_send_ex 函数处理，next 和 send 的等价性也源于此。经过千辛万苦，我们终于找到了理解生成器运行机制的关键所在。\n",
    "\n",
    "gen_send_ex 函数同样位于 Objects/genobject.c 源文件，函数挺长，但最关键的代码只有两行：\n",
    "\n",
    "    f->f_back = tstate->frame;\n",
    "\n",
    "    // ...\n",
    "\n",
    "    result = PyEval_EvalFrameEx(f, exc);\n",
    "    \n",
    "首先，第一行代码将生成器栈帧挂到当前调用链上；然后，第二行代码调用 PyEval_EvalFrameEx 开始执行生成器栈帧；生成器栈帧对象保存着生成器执行上下文，其中 f_lasti 字段跟踪生成器代码对象的执行进度。\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_32.jpg\" width=800 height=600> \n",
    "\n",
    "PyEval_EvalFrameEx 函数最终调用 _PyEval_EvalFrameDefault 函数执行 frame 对象上的代码对象。这个函数我们在虚拟机部分学习过，对它并不陌生。虽然它体量巨大，超过 3 千行代码，逻辑却非常直白 —— 内部由无限 for 循环逐条遍历并处理字节码，每执行完一条字节码就自增 f_lasti 字段。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hired-drove",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 生成器的暂停"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accepting-terrorism",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们知道，生成器可以利用 yield 语句，将执行权归还给调用者。因此，生成器暂停执行的秘密就隐藏在 yield 语句中。我们先来看看 yield 语句编译后，生成什么字节码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "hundred-vector",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:08:18.251025Z",
     "start_time": "2021-03-30T07:08:18.238034Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  2           0 LOAD_GLOBAL              0 (print)\n",
      "              2 LOAD_CONST               1 ('task with argument {} started')\n",
      "              4 LOAD_METHOD              1 (format)\n",
      "              6 LOAD_FAST                0 (arg)\n",
      "              8 CALL_METHOD              1\n",
      "             10 CALL_FUNCTION            1\n",
      "             12 POP_TOP\n",
      "\n",
      "  4          14 LOAD_CONST               2 (1)\n",
      "             16 YIELD_VALUE\n",
      "             18 STORE_FAST               1 (data)\n",
      "\n",
      "  5          20 LOAD_GLOBAL              0 (print)\n",
      "             22 LOAD_CONST               3 ('step one finished， got {} from caller')\n",
      "             24 LOAD_METHOD              1 (format)\n",
      "             26 LOAD_FAST                1 (data)\n",
      "             28 CALL_METHOD              1\n",
      "             30 CALL_FUNCTION            1\n",
      "             32 POP_TOP\n",
      "\n",
      "  7          34 LOAD_CONST               4 (2)\n",
      "             36 YIELD_VALUE\n",
      "             38 STORE_FAST               1 (data)\n",
      "\n",
      "  8          40 LOAD_GLOBAL              0 (print)\n",
      "             42 LOAD_CONST               5 ('step two finished， got {} from caller')\n",
      "             44 LOAD_METHOD              1 (format)\n",
      "             46 LOAD_FAST                1 (data)\n",
      "             48 CALL_METHOD              1\n",
      "             50 CALL_FUNCTION            1\n",
      "             52 POP_TOP\n",
      "\n",
      " 10          54 LOAD_CONST               6 (3)\n",
      "             56 YIELD_VALUE\n",
      "             58 STORE_FAST               1 (data)\n",
      "\n",
      " 11          60 LOAD_GLOBAL              0 (print)\n",
      "             62 LOAD_CONST               7 ('step three finished， got {} from caller')\n",
      "             64 LOAD_METHOD              1 (format)\n",
      "             66 LOAD_FAST                1 (data)\n",
      "             68 CALL_METHOD              1\n",
      "             70 CALL_FUNCTION            1\n",
      "             72 POP_TOP\n",
      "             74 LOAD_CONST               0 (None)\n",
      "             76 RETURN_VALUE\n"
     ]
    }
   ],
   "source": [
    ">>> import dis\n",
    ">>> dis.dis(co_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secret-shanghai",
   "metadata": {
    "hidden": true
   },
   "source": [
    "可以看到，co_process 每个 yield 语句编译后，都得到这样 3 条字节码指令：\n",
    "\n",
    "      4          14 LOAD_CONST               2 (1)\n",
    "                 16 YIELD_VALUE\n",
    "                 18 STORE_FAST               1 (data)\n",
    "                 \n",
    " 首先，LOAD_CONST 将需要带给调用者的值 ( yield 右边的表达式值) 加载到运行栈栈顶：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_33.jpg\" width=800 height=600> \n",
    " \n",
    " 接着，YIELD_VALUE 开始放大招，它先从栈顶弹出 yield 值作为 _PyEval_EvalFrameDefault 函数返回值，然后一个 goto 语句跳出 for 循环，干净利落：\n",
    " \n",
    "         TARGET(YIELD_VALUE) {\n",
    "            retval = POP();\n",
    "\n",
    "            if (co->co_flags & CO_ASYNC_GENERATOR) {\n",
    "                PyObject *w = _PyAsyncGenValueWrapperNew(retval);\n",
    "                Py_DECREF(retval);\n",
    "                if (w == NULL) {\n",
    "                    retval = NULL;\n",
    "                    goto error;\n",
    "                }\n",
    "                retval = w;\n",
    "            }\n",
    "\n",
    "            f->f_stacktop = stack_pointer;\n",
    "            why = WHY_YIELD;\n",
    "            goto fast_yield;\n",
    "        }\n",
    "\n",
    "紧接着，_PyEval_EvalFrameDefault 函数将当前栈帧 (也就是生成器的栈帧) 从调用链中解开。注意到，yield 值被 _PyEval_EvalFrameDefault 函数返回，并最终被 send 方法或 next 函数返回给调用者。\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_34.jpg\" width=800 height=600> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exempt-proof",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 生成器的恢复"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "european-albania",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当我们再次调用 send 方法时，生成器将恢复执行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "another-manner",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:14:00.040931Z",
     "start_time": "2021-03-30T07:14:00.021923Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step one finished， got hello from caller\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> genco.send('hello')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "starting-script",
   "metadata": {
    "hidden": true
   },
   "source": [
    "注意到，通过 send 发送的数据作为 yield 语句的值，被生成器获取并保存在局部变量 data ，进而被输出。这一切是如何做到的呢？我们接着分析。\n",
    "\n",
    "我们知道，send 方法被调用后，Python 先把生成器栈帧对象挂到调用链，并最终调用 PyEval_EvalFrameEx 函数逐条执行字节码。在这个过程中，send 发送的数据会被放在生成器栈顶：\n",
    "\n",
    " <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_35.jpg\" width=800 height=600> \n",
    " \n",
    " 生成器执行进度被保存在 f_lasti 字段，生成器将从下一条字节码指令 STORE_FAST 继续执行。STORE_FAST 指令从栈顶取出 send 发来的数据，并保存到局部变量 data ：\n",
    " \n",
    "  <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_35.jpg\" width=800 height=600> \n",
    "\n",
    "再接着，生成器将按照正常的逻辑，有条不紊地执行，直到遇到下一个 yield 语句或者生成器函数返回。\n",
    "\n",
    "至此，生成器执行、暂停、恢复的全部秘密皆已揭开！\n",
    "\n",
    "- 生成器函数编译后代码对象带有 CO_GENERATOR 标识；\n",
    "- 如果函数代码对象带 CO_GENERATOR 标识，被调用时 Python 将创建生成器对象；\n",
    "- 生成器创建的同时，Python 还创建一个栈帧对象，用于维护代码对象执行上下文；\n",
    "- 调用 next/send 驱动生成器执行，Python 将生成器栈帧对象接入调用链，开始执行字节码；\n",
    "- 执行到 yield 语句时，Python 将 yield 右边的值放入栈顶，并结束字节码执行循环，执行权回到上一个栈帧；\n",
    "- yield 值最终作为 next 函数或 send 方法的返回值，被调用者取得；\n",
    "- 再次调用 *next/*send ，Python 重新将生成器栈帧对象接入调用链恢复执行，通过 send 发送的值被放在栈顶；\n",
    "- 生成器函数重新启动后，从 YIELD_VALUE 后的字节码恢复执行，可从栈顶获得调用者发来的值；\n",
    "- 代码执行权就这样在调用者和生成器间来回切换，而生成器栈顶被用来传值；\n",
    "\n",
    "生成器运行机制全面解密后，我们便可以利用这些特性来实现协程。下一节，我们将通过自己的双手，实现一个完整的协程库！这个玩具代码量仅 100 来行，却揭示了协程库的核心机密。经过这次实战，我们将彻底驾驭协程，在协程应用开发中更加游刃有余。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "reverse-furniture",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:36:04.297181Z",
     "start_time": "2021-03-30T07:36:04.283189Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object co_process at 0x000002A6DF92EAC0>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "co_process('foo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "listed-harvey",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:37:15.900850Z",
     "start_time": "2021-03-30T07:37:15.893844Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "genco = co_process('foo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "unable-mixture",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:37:25.744257Z",
     "start_time": "2021-03-30T07:37:25.729251Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "task with argument foo started\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(genco)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "vietnamese-representation",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:37:33.493646Z",
     "start_time": "2021-03-30T07:37:33.479655Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step one finished， got None from caller\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(genco)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "three-westminster",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:38:29.933375Z",
     "start_time": "2021-03-30T07:38:29.922382Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step two finished， got Go from caller\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genco.send('Go')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "substantial-shelf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T07:38:36.417986Z",
     "start_time": "2021-03-30T07:38:36.407982Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step three finished， got GoGo from caller\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-65-86a4a1f83d95>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mgenco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;34m'GoGo'\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: "
     ]
    }
   ],
   "source": [
    "genco.send('GoGo')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atomic-synthesis",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 再造轮子：手把手教你设计协程库"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exceptional-nancy",
   "metadata": {
    "hidden": true
   },
   "source": [
    "经过前面章节学习，我们知道生成器可以通过 yield 将执行权还给调用者。这时生成器会记住自己的执行进度，重新调度后将恢复执行。利用这个特性，我们可以实现用户态 协程 。但具体如何实施呢？\n",
    "\n",
    "本节，我们将在 Python 交互式终端上探索协程库的设计思路，最终形成一个精简的协程库，代码量仅 100 来行！麻雀虽小，五脏俱全！借此即可彻底掌握 **协程运行原理** 和 **协程库设计精髓** ，玩转协程式应用开发。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decreased-manor",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 事件循环建模"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "promising-gross",
   "metadata": {
    "hidden": true
   },
   "source": [
    "简而言之，我们需要实现一个 **事件循环** ( Event Loop )，它内部有一个 **可执行** ( Runnable ) 协程队列：\n",
    "\n",
    "  <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_37.jpg\" width=800 height=600> \n",
    "\n",
    "事件循环是一个永久循环，每次循环时它先调度可执行队列里的每个协程 —— 即从队列中取出一个可执行协程，然后调用 send 方法驱动它执行：\n",
    "\n",
    "  <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_38.jpg\" width=800 height=600> \n",
    "  \n",
    " 协程执行的结果可分为两种不同情况。其一，协程没有遇到 IO 操作，一把梭哈到底并最后退出。这时， send 方法抛 StopIteration 异常通知调用者：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_39.jpg\" width=800 height=600> \n",
    "\n",
    "其二，协程需要进行 IO 操作，这时它应该通过 yield 让出执行权，并将 IO 操作上下文提交给事件循环。IO 操作由事件循环负责执行，操作上下文必须记录协程信息：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_40.jpg\" width=800 height=600> \n",
    "可执行队列处理完毕后，得到成一个个 IO 操作上下文，事件循环负责将它们注册到 epoll ，以便订阅 IO 事件：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_41.jpg\" width=800 height=600> \n",
    "接着，事件循环通过 epoll 等待 IO 事件到达。当某个 IO 操作就绪时，事件循环将把对应协程重新放入可执行队列。假设协程 3 等待的 IO 操作已经就绪，epoll 将返回对应 IO 事件，执行 IO 处理函数并将协程放回可执行队列重新调度：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_42.jpg\" width=800 height=600> \n",
    "事件循环处理完所有 epoll 事件后，将进入下一次循环。这时，又开始处理可执行队列，周而复始。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "velvet-carbon",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:17:59.263950Z",
     "start_time": "2021-03-30T08:17:59.255954Z"
    },
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### epoll"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "earned-fault",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于事件循环需要同时关注多个 IO 操作，因此需要采用 IO **多路复用** 技术。那么，什么是 IO 多路复用呢？它又是如何使用的呢？epoll 是 Linux 下的 IO 多路复用技术，很有代表性。我们便以 epoll 为例，简单探讨一下。\n",
    "\n",
    "服务器应用一般需要通过 **套接字** ( socket ) 监听某个端口，等待客户端连接。这个函数用于创建一个监听套接字： \n",
    "\n",
    "     from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR\n",
    "\n",
    "    def create_listen_socket(bind_addr='0.0.0.0', bind_port=55555, backlogs=102400):\n",
    "        # 创建套接字\n",
    "        sock = socket(AF_INET, SOCK_STREAM)\n",
    "        # 设置地址复用选项\n",
    "        sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)\n",
    "        # 绑定监听地址和端口\n",
    "        sock.bind((bind_addr, bind_port))\n",
    "        # 开始监听\n",
    "        sock.listen(backlogs)\n",
    "\n",
    "        return sock\n",
    "我们以默认参数创建一个监听套接字，并调用 accept 方法接受客户端连接：\n",
    "\n",
    "    >>> s = create_listen_socket()\n",
    "    >>> s.accept()\n",
    "\n",
    "accept 调用将 **阻塞** ，直到有客户端连接上来才会返回。现在，我们通过 telnet 命令模拟客户端连接：\n",
    "\n",
    "    $ telnet 127.0.0.1 55555\n",
    "    \n",
    "当客户端连上来后，accept 调用就返回了，返回值是一个元组。元组包含一个与客户端通讯的套接字，以及客户端的地址端口对信息：\n",
    "\n",
    "    >>> s.accept()\n",
    "    (<socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 55555), raddr=('127.0.0.1', 41990)>, ('127.0.0.1', 41990))\n",
    "    \n",
    "如果程序还需要处理另一个监听套接字，事情就尴尬了。假设我们在 s 上等待客户端连接，这时 accept 将阻塞；就算 s2 套接字上来了新连接，也无法提前返回：\n",
    "\n",
    "    >>> s2 = create_listen_socket(bind_port=44444)\n",
    "    >>> s.accept()\n",
    "    \n",
    "这该怎么办呢？我们先把套接字设置成 非阻塞 状态，accept 就不会一直阻塞了：\n",
    "\n",
    "    >>> s.setblocking(False)\n",
    "    >>> s.accept()\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"/home/fasion/opt/python3.8.5/lib/python3.8/socket.py\", line 292, in accept\n",
    "        fd, addr = self._accept()\n",
    "    BlockingIOError: [Errno 11] Resource temporarily unavailable\n",
    "\n",
    "由于 s 套接字上没有新连接，accept 将抛出 BlockingIOError 异常，以此告知调用者。这时，我们就可以抽出身来处理 s2 了。如果 s2 也没有新连接了，我们又再次检查 s 。\n",
    "\n",
    "    >>> s2.accept()\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"/home/fasion/opt/python3.8.5/lib/python3.8/socket.py\", line 292, in accept\n",
    "        fd, addr = self._accept()\n",
    "    BlockingIOError: [Errno 11] Resource temporarily unavailable\n",
    "    >>> s.accept()\n",
    "    Traceback (most recent call last):\n",
    "      File \"<stdin>\", line 1, in <module>\n",
    "      File \"/home/fasion/opt/python3.8.5/lib/python3.8/socket.py\", line 292, in accept\n",
    "        fd, addr = self._accept()\n",
    "    BlockingIOError: [Errno 11] Resource temporarily unavailable\n",
    "\n",
    "最终，我们将在 s 和 s2 间来回 轮询 ，但轮询很浪费 CPU 资源！特别是套接字很多时，更是如此！如果能让内核同时关注多个套接字，当它们中有新连接达到时再通知我们就好了 —— 这就是 epoll 擅长的事。\n",
    "\n",
    "当监听套接字上有新连接时，它会产生 **读事件** 。因此，我们可以创建一个 epoll 描述符，并将 s 和 s2 注册进去，订阅 **读事件** ( EPOLLIN )：\n",
    "\n",
    "    >>> import select\n",
    "    >>> ep = select.epoll()\n",
    "    >>> ep.register(s.fileno(), select.EPOLLIN)\n",
    "    >>> ep.register(s2.fileno(), select.EPOLLIN)\n",
    "    \n",
    "接着，我们调用 poll 方法，等待我们感兴趣的事件：\n",
    "\n",
    "    >>> events = ep.poll()\n",
    "\n",
    "poll 将一直阻塞，直到 s 或 s2 上有新连接达到。试着连一下 s2 ：\n",
    "\n",
    "    $ telnet 127.0.0.1 44444\n",
    "    \n",
    "poll 立马停止阻塞，并向我们返回了一个事件列表，列表项是一个由 **文件描述符** 和 **事件掩码** 组成的元组：\n",
    "\n",
    "    >>> events\n",
    "    [(6, 1)]\n",
    "    >>> for fileno, event in events:\n",
    "    ...     print(fileno, event)\n",
    "    ...\n",
    "    6 1\n",
    "    \n",
    "这个信息告诉我们，哪个套接字上有什么事件发生。如此一来，程序可以精准处理套接字，无须傻傻 **轮询** 。这就是 epoll 的强大能力，它让高效处理大规模套接字成为可能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "representative-street",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 调度第一个协程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wrapped-agenda",
   "metadata": {
    "hidden": true
   },
   "source": [
    "开始研究有 IO 操作的协程之前，我们先拿一个纯计算协程练练手。这是一个只做加法运算的协程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "passing-national",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:26:33.240810Z",
     "start_time": "2021-03-30T08:26:33.230816Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def add(a, b):\n",
    "    if False:\n",
    "        yield\n",
    "    return a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broken-edmonton",
   "metadata": {
    "hidden": true
   },
   "source": [
    "if 语句永远不会执行，它只是为了引入 yield 语句，让 Python 将 add 编译成生成器。\n",
    "\n",
    "现在我们创建一个新协程，并调用 send 方法把它调度起来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "civil-integration",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:27:05.656876Z",
     "start_time": "2021-03-30T08:27:05.639885Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object add at 0x000002A6DFB025F0>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "co = add(1, 2)\n",
    "co"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "human-belly",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:27:17.494524Z",
     "start_time": "2021-03-30T08:27:17.484535Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "3",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-69-cf42f0b7f3cb>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: 3"
     ]
    }
   ],
   "source": [
    "co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "modular-latex",
   "metadata": {
    "hidden": true
   },
   "source": [
    "正如前面提到的那样，协程一把梭哈到底，StopIteration 异常告诉我们它已经执行完毕，结果是 3 。注意到，协程执行结果 (函数返回值) 保存在 StopIteration 的 value 属性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "elder-alfred",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:28:03.195366Z",
     "start_time": "2021-03-30T08:28:03.183369Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(StopIteration(3), 3)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "e = sys.last_value\n",
    "e, e.value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "revolutionary-sunrise",
   "metadata": {
    "hidden": true
   },
   "source": [
    "我们还可以写一个函数来调度协程，函数只需调用 send 方法，并在协程执行完毕后输出一些提示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "altered-savage",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:28:24.170923Z",
     "start_time": "2021-03-30T08:28:24.155917Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def schedule_coroutine(co):\n",
    "    try:\n",
    "        co.send(None)\n",
    "    except StopIteration as e:\n",
    "        print('coroutine {} completed with result {}'.format(co.__name__, e.value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "cultural-marathon",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:28:39.404143Z",
     "start_time": "2021-03-30T08:28:39.400144Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "coroutine add completed with result 5\n"
     ]
    }
   ],
   "source": [
    "co = add(2, 3)\n",
    "schedule_coroutine(co)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sufficient-garage",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### IO 上下文"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "minus-silly",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果协程中涉及 IO 操作，则需要在 IO 未就绪时通过 yield 让出执行权。在让出执行权的同时，还需要将 IO 上下文提交给事件循环，由它协助处理。那么，IO 上下文需要包含哪些信息呢？\n",
    "\n",
    "IOContext 需要保存哪些信息取决于封装程度，但至少要包括协程需要等待的 **文件描述符** 以及感兴趣的 **事件** ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "literary-italic",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:29:17.913577Z",
     "start_time": "2021-03-30T08:29:17.909593Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class IOContext:\n",
    "    \n",
    "    def __init__(self, fileno, events):\n",
    "        self.fileno = fileno\n",
    "        self.events = events"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extensive-accident",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在我们开始编写一个带 IO 操作的协程，它负责从监听套接字接收新客户端连接："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "textile-official",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:29:30.611672Z",
     "start_time": "2021-03-30T08:29:30.605677Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def accept_client(sock):\n",
    "    while True:\n",
    "        try:\n",
    "            return sock.accept()\n",
    "        except BlockingIOError:\n",
    "            pass\n",
    "        \n",
    "        yield IOContext(sock.fileno(), select.EPOLLIN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "micro-herald",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程主体逻辑是一个循环，它先调用 accept 尝试接收新连接。如果没有连接就绪，accept 会抛 BlockingIOError 异常。 这时，yield 语句让出执行权，并将 IOContext 提交给事件循环。注意到，协程对套接字上的读事件感兴趣。\n",
    "\n",
    "现在我们创建一个这样的协程，并扮演事件循环，来体会协程调度过程。如果套接字 s 没有就绪连接，send 将收到协程返回的 IOContext ，表明协程期待哪些事件发生：\n",
    "\n",
    "    >>> co = accept_client(s)\n",
    "    >>> context = co.send(None)\n",
    "    >>> context\n",
    "    <__main__.IOContext object at 0x7fcd58e3ef70>\n",
    "    >>> context.fileno\n",
    "    3\n",
    "    >>> context.events\n",
    "    1\n",
    "    \n",
    "事件循环接到上下文后，需要将当前协程保存到上下文中，并将需要订阅的事件注册到 epoll ：\n",
    "\n",
    "    >>> context.co = co\n",
    "    >>> ep.register(context.fileno, context.events)\n",
    "    \n",
    "接着，事件循环在 epoll 上等待相关事件到达：\n",
    "\n",
    "    >>> ep.poll()\n",
    "    \n",
    "poll 将保持阻塞，直到有注册事件出现。因此，用 telnet 命令再次连接 s 套接字，poll 将返回：\n",
    "\n",
    "    >>> ep.poll()\n",
    "    [(3, 1)]\n",
    "    \n",
    "根据 poll 返回的文件描述符 3 ，我们知道 context 这次 IO 操作已经就绪了。这时，可以接着调度对应的协程：\n",
    "\n",
    "    >>> context.co.send(None)\n",
    "    Traceback (most recent call last):\n",
    "    File \"<stdin>\", line 1, in <module>\n",
    "    StopIteration: (<socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 55555), raddr=('127.0.0.1', 51528)>, ('127.0.0.1', 51528)) \n",
    "    \n",
    "至此，协程 co 成功接收了一个客户端连接，并退出了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surprised-dividend",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### yield from"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "humanitarian-pregnancy",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现有一个用于计算圆面积的协程，它没有涉及 IO 操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "prompt-berkeley",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:37:00.611073Z",
     "start_time": "2021-03-30T08:37:00.605075Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def circle_area(r):\n",
    "    if False:\n",
    "        yield\n",
    "    return math.pi * r ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "packed-withdrawal",
   "metadata": {
    "hidden": true
   },
   "source": [
    "创建一个这样的协程来计算半径为 2 的圆的面积，并调用 send 方法来调度它，协程执行完毕后将返回结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "senior-sector",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:37:09.296245Z",
     "start_time": "2021-03-30T08:37:09.288264Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> co = circle_area(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "purple-header",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:37:13.838367Z",
     "start_time": "2021-03-30T08:37:13.818378Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "12.566370614359172",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-79-cf42f0b7f3cb>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: 12.566370614359172"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advance-optics",
   "metadata": {
    "hidden": true
   },
   "source": [
    "现在，让我们利用这个协程来计算圆柱体积："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "included-cyprus",
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def cylindrical_volume(r, h):\n",
    "    return circle_area(r) * h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "checked-helen",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这样显然是不行的，因为调用 circle_area 返回的是一个代表协程的生成器，需要调度它才能获得计算结果。不过没关系，我们可以这么写："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "still-pencil",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:38:01.953991Z",
     "start_time": "2021-03-30T08:38:01.946988Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def cylindrical_volume(r, h):\n",
    "    co = circle_area(r)\n",
    "    while True:\n",
    "        try:\n",
    "            yield co.send(None)\n",
    "        except StopIteration as e:\n",
    "            floorage = e.value\n",
    "            return floorage * h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blank-hypothesis",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这个是一个协程函数，它先创建一个子协程用于计算底面积，然后用一个永久循环驱动子协程执行。\n",
    "\n",
    "每次循环时，它先调用 send 方法将执行权交给子协程。如果子协程用 yield 语句归还执行权，这里同样用 yield 将执行权交给调用者，yield 值也一并向上传递。如果子协程退出，它将取出子协程执行结果并完成计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "forward-invasion",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:38:32.063912Z",
     "start_time": "2021-03-30T08:38:32.040925Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "37.69911184307752",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-81-fc96f0c4b5e5>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m      1\u001B[0m \u001B[0mco\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mcylindrical_volume\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;36m2\u001B[0m\u001B[1;33m,\u001B[0m \u001B[1;36m3\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 2\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: 37.69911184307752"
     ]
    }
   ],
   "source": [
    "co = cylindrical_volume(2, 3)\n",
    "co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wicked-bryan",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:38:53.029328Z",
     "start_time": "2021-03-30T08:38:53.020323Z"
    },
    "hidden": true
   },
   "source": [
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_05\\C5_43.jpg\" width=800 height=600> \n",
    "因此，cylindrical_volume 就像一个中间人，在调用者和子协程之间来回传递执行权。函数调用很常见，如果涉及协程的函数调用都需要用样板代码传递执行权，那简直就是一个噩梦！为此，Python 引入 yield from ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "backed-lecture",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:41:09.781240Z",
     "start_time": "2021-03-30T08:41:09.779240Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def cylindrical_volume(r, h):\n",
    "    floorage = yield from circle_area(r)\n",
    "    return floorage * h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "together-directive",
   "metadata": {
    "hidden": true
   },
   "source": [
    "例子中 yield from 的作用相当于上一例子中的 while 循环，因此这两个例子是完全等价的。与业务逻辑无关的样板代码消除后，新函数变得简洁纯粹，更加清晰易懂了！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "tracked-impossible",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:41:43.342507Z",
     "start_time": "2021-03-30T08:41:43.322520Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "37.69911184307752",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-84-fc96f0c4b5e5>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m      1\u001B[0m \u001B[0mco\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mcylindrical_volume\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;36m2\u001B[0m\u001B[1;33m,\u001B[0m \u001B[1;36m3\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m----> 2\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: 37.69911184307752"
     ]
    }
   ],
   "source": [
    "co = cylindrical_volume(2, 3)\n",
    "co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rising-impact",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### async await"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incredible-sierra",
   "metadata": {
    "hidden": true
   },
   "source": [
    "直接使用生成器实现协程，虽然逻辑上可行，但语义上有点令人摸不着头脑："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "medical-checkout",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:44:15.798227Z",
     "start_time": "2021-03-30T08:44:15.784236Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object circle_area at 0x000002A6DEEE8970>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> co = circle_area(1)\n",
    ">>> co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "direct-geometry",
   "metadata": {
    "hidden": true
   },
   "source": [
    "为突显协程语义，Python 引入了 async 关键字："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "continent-string",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:44:27.065230Z",
     "start_time": "2021-03-30T08:44:27.057253Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "async def circle_area(r):\n",
    "    return math.pi * r ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "packed-mouse",
   "metadata": {
    "hidden": true
   },
   "source": [
    "被 async 关键字标识的函数会被编译成异步函数，调用后得到一个 coroutine 对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "legitimate-identity",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:44:42.454556Z",
     "start_time": "2021-03-30T08:44:42.441549Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<coroutine object circle_area at 0x000002A6DFB682C0>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> co = circle_area(1)\n",
    ">>> co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "generic-procedure",
   "metadata": {
    "hidden": true
   },
   "source": [
    "coroutine 对象与 generator 对象类似，我们可以调用 send 方法来调度 coroutine 对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "surrounded-command",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:44:58.008198Z",
     "start_time": "2021-03-30T08:44:57.990208Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "StopIteration",
     "evalue": "3.141592653589793",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-88-cf42f0b7f3cb>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: 3.141592653589793"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "parallel-terror",
   "metadata": {
    "hidden": true
   },
   "source": [
    "coroutine 对象的语义更加准确，而且我们再也不需要在函数代码中显式编写 yield 语句了，这未免有点画蛇添足。\n",
    "\n",
    "青出于蓝而胜于蓝，如果 coroutine 没执行完毕便被意外销毁，Python 将输出警告信息："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "virgin-cartridge",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:45:15.776274Z",
     "start_time": "2021-03-30T08:45:15.764282Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-89-48a23e497293>:2: RuntimeWarning: coroutine 'circle_area' was never awaited\n",
      "  del co\n",
      "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n"
     ]
    }
   ],
   "source": [
    ">>> co = circle_area(2)\n",
    ">>> del co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strange-hypothetical",
   "metadata": {
    "hidden": true
   },
   "source": [
    "此外，Python 还引入了 await 关键字，代替前面提到的 yield from 语句。与 yield from 类似，await 将执行权交给子协程，并等待它退出。如果子协程需要暂时归还执行权，await 同样承担起中间人角色，在调用者与子协程间来回接棒。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "reasonable-campbell",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:45:49.887921Z",
     "start_time": "2021-03-30T08:45:49.881938Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "async def cylindrical_volume(r, h):\n",
    "    floorage = await circle_area(r)\n",
    "    return floorage * h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affected-cache",
   "metadata": {
    "hidden": true
   },
   "source": [
    "无须多言，await 的语义也比 yield from 准确。另外，Python 还引入了 可等待对象 ( awaitable )。例子如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "opposite-executive",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:46:11.489348Z",
     "start_time": "2021-03-30T08:46:11.486348Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class Job:\n",
    "    \n",
    "    def __await__(self):\n",
    "        print('step 1')\n",
    "        yield\n",
    "        print('step 2')\n",
    "        yield\n",
    "        print('step 3')\n",
    "        return 'coding-fan'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "informal-charger",
   "metadata": {
    "hidden": true
   },
   "source": [
    "可等待对象需要提供 __await__ 魔术方法，实现成普通生成器即可。然后，await 就可以驱动生成器的执行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "assured-organic",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:46:25.686231Z",
     "start_time": "2021-03-30T08:46:25.678235Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "async def do_job(job):\n",
    "    value = await job\n",
    "    print('job is done with value {}'.format(value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "intelligent-leeds",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:47:19.109360Z",
     "start_time": "2021-03-30T08:47:19.096369Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> co = do_job(Job())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "stunning-martin",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:47:25.233837Z",
     "start_time": "2021-03-30T08:47:25.219858Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1\n"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "framed-ready",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:47:30.636401Z",
     "start_time": "2021-03-30T08:47:30.622410Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 2\n"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "blocked-medline",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:47:38.122907Z",
     "start_time": "2021-03-30T08:47:38.113903Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 3\n",
      "job is done with value coding-fan\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-99-cf42f0b7f3cb>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: "
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mature-inspection",
   "metadata": {
    "hidden": true
   },
   "source": [
    "如果你觉得可等待对象 Job 难以理解，可以将它想象成等价的 yield from 形式，便豁然开朗了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "enabling-enhancement",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:48:05.292516Z",
     "start_time": "2021-03-30T08:48:05.277525Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def do_job(job):\n",
    "    value = yield from job.__await__()\n",
    "    print('job is done with value {}'.format(value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "heated-avatar",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:48:27.275651Z",
     "start_time": "2021-03-30T08:48:27.260649Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    ">>> co = do_job(Job())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "removable-yemen",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:48:33.572972Z",
     "start_time": "2021-03-30T08:48:33.557966Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1\n"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "subjective-istanbul",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:48:41.921716Z",
     "start_time": "2021-03-30T08:48:41.905727Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 2\n"
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "headed-laundry",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:48:42.700126Z",
     "start_time": "2021-03-30T08:48:42.680138Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 3\n",
      "job is done with value coding-fan\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m                             Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-104-cf42f0b7f3cb>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0mco\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0msend\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;32mNone\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mStopIteration\u001B[0m: "
     ]
    }
   ],
   "source": [
    ">>> co.send(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "charming-rough",
   "metadata": {
    "hidden": true
   },
   "source": [
    "await 本无法驱动普通生成器，可等待对象却另辟蹊径，因而它在协程库中有重要作用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "champion-residence",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 终极作品"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opening-destruction",
   "metadata": {
    "hidden": true
   },
   "source": [
    "铺垫了这么东西，终于可以亮出我们的终极作品了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "alike-racing",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T08:49:05.928157Z",
     "start_time": "2021-03-30T08:49:05.899186Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "module 'select' has no attribute 'epoll'",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mAttributeError\u001B[0m                            Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-105-1b75428bc040>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[0;32m    159\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m    160\u001B[0m \u001B[1;32mif\u001B[0m \u001B[0m__name__\u001B[0m \u001B[1;33m==\u001B[0m \u001B[1;34m'__main__'\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m--> 161\u001B[1;33m     \u001B[0mmain\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;32m<ipython-input-105-1b75428bc040>\u001B[0m in \u001B[0;36mmain\u001B[1;34m()\u001B[0m\n\u001B[0;32m    154\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m    155\u001B[0m \u001B[1;32mdef\u001B[0m \u001B[0mmain\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m--> 156\u001B[1;33m     \u001B[0mloop\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mEventLoop\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m    157\u001B[0m     \u001B[0mserver\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mTcpServer\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mloop\u001B[0m\u001B[1;33m=\u001B[0m\u001B[0mloop\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m    158\u001B[0m     \u001B[0mloop\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mrun_forever\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;32m<ipython-input-105-1b75428bc040>\u001B[0m in \u001B[0;36m__init__\u001B[1;34m(self)\u001B[0m\n\u001B[0;32m     84\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m     85\u001B[0m     \u001B[1;32mdef\u001B[0m \u001B[0m__init__\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mself\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m:\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[1;32m---> 86\u001B[1;33m         \u001B[0mself\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mepoll\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mselect\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mepoll\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m\u001B[0;32m     87\u001B[0m \u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0;32m     88\u001B[0m         \u001B[0mself\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mrunnables\u001B[0m \u001B[1;33m=\u001B[0m \u001B[0mdeque\u001B[0m\u001B[1;33m(\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n",
      "\u001B[1;31mAttributeError\u001B[0m: module 'select' has no attribute 'epoll'"
     ]
    }
   ],
   "source": [
    "import select\n",
    "\n",
    "from collections import deque\n",
    "from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR\n",
    "\n",
    "def create_listen_socket(bind_addr='0.0.0.0', bind_port=55555, backlogs=102400):\n",
    "    sock = socket(AF_INET, SOCK_STREAM)\n",
    "    sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)\n",
    "    sock.bind((bind_addr, bind_port))\n",
    "    sock.listen(backlogs)\n",
    "    return sock\n",
    "\n",
    "class Future:\n",
    "\n",
    "    def __init__(self, loop):\n",
    "        self.loop = loop\n",
    "        self.done = False\n",
    "        self.result = None\n",
    "        self.co = None\n",
    "\n",
    "    def set_coroutine(self, co):\n",
    "        self.co = co\n",
    "\n",
    "    def set_result(self, result):\n",
    "        self.done = True\n",
    "        self.result = result\n",
    "\n",
    "        if self.co:\n",
    "            self.loop.add_coroutine(self.co)\n",
    "\n",
    "    def __await__(self):\n",
    "        if not self.done:\n",
    "            yield self\n",
    "        return self.result\n",
    "\n",
    "class AsyncSocket:\n",
    "\n",
    "    def __init__(self, sock, loop):\n",
    "        sock.setblocking(False)\n",
    "\n",
    "        self.sock = sock\n",
    "        self.loop = loop\n",
    "\n",
    "    def fileno(self):\n",
    "        return self.sock.fileno()\n",
    "\n",
    "    def create_future_for_events(self, events):\n",
    "        future = self.loop.create_future()\n",
    "\n",
    "        def handler(fileno, active_events):\n",
    "            self.loop.unregister_from_polling(self.fileno())\n",
    "            future.set_result(active_events)\n",
    "\n",
    "        self.loop.register_for_polling(self.fileno(), events, handler)\n",
    "\n",
    "        return future\n",
    "\n",
    "    async def accept(self):\n",
    "        while True:\n",
    "            try:\n",
    "                sock, addr = self.sock.accept()\n",
    "                return AsyncSocket(sock=sock, loop=self.loop), addr\n",
    "            except BlockingIOError:\n",
    "                future = self.create_future_for_events(select.EPOLLIN)\n",
    "                await future\n",
    "\n",
    "    async def recv(self, bufsize):\n",
    "        while True:\n",
    "            try:\n",
    "                return self.sock.recv(bufsize)\n",
    "            except BlockingIOError:\n",
    "                future = self.create_future_for_events(select.EPOLLIN)\n",
    "                await future\n",
    "\n",
    "    async def send(self, data):\n",
    "        while True:\n",
    "            try:\n",
    "                return self.sock.send(data)\n",
    "            except BlockingIOError:\n",
    "                future = self.create_future_for_events(select.EPOLLOUT)\n",
    "                await future\n",
    "\n",
    "class EventLoop:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.epoll = select.epoll()\n",
    "\n",
    "        self.runnables = deque()\n",
    "        self.handlers = {}\n",
    "\n",
    "    def create_future(self):\n",
    "        return Future(loop=self)\n",
    "\n",
    "    def create_listen_socket(self, bind_addr, bind_port, backlogs=102400):\n",
    "        sock = create_listen_socket(bind_addr, bind_port, backlogs)\n",
    "        return AsyncSocket(sock=sock, loop=self)\n",
    "\n",
    "    def register_for_polling(self, fileno, events, handler):\n",
    "        print('register fileno={} for events {}'.format(fileno, events))\n",
    "        self.handlers[fileno] = handler\n",
    "        self.epoll.register(fileno, events)\n",
    "\n",
    "    def unregister_from_polling(self, fileno):\n",
    "        print('unregister fileno={}'.format(fileno))\n",
    "        self.epoll.unregister(fileno)\n",
    "        self.handlers.pop(fileno)\n",
    "\n",
    "    def add_coroutine(self, co):\n",
    "        self.runnables.append(co)\n",
    "\n",
    "    def run_coroutine(self, co):\n",
    "        try:\n",
    "            future = co.send(None)\n",
    "            future.set_coroutine(co)\n",
    "        except StopIteration as e:\n",
    "            print('coroutine {} stopped'.format(co.__name__))\n",
    "\n",
    "    def schedule_runnable_coroutines(self):\n",
    "        while self.runnables:\n",
    "            self.run_coroutine(co=self.runnables.popleft())\n",
    "\n",
    "    def run_forever(self):\n",
    "        while True:\n",
    "            self.schedule_runnable_coroutines()\n",
    "\n",
    "            events = self.epoll.poll(1)\n",
    "            for fileno, event in events:\n",
    "                handler = self.handlers.get(fileno)\n",
    "                if handler:\n",
    "                    handler(fileno, events)\n",
    "\n",
    "class TcpServer:\n",
    "\n",
    "    def __init__(self, loop, bind_addr='0.0.0.0', bind_port=55555):\n",
    "        self.loop = loop\n",
    "        self.listen_sock = self.loop.create_listen_socket(bind_addr=bind_addr, bind_port=bind_port)\n",
    "        self.loop.add_coroutine(self.serve_forever())\n",
    "\n",
    "    async def serve_client(self, sock):\n",
    "        while True:\n",
    "            data = await sock.recv(1024)\n",
    "            if not data:\n",
    "                print('client disconnected')\n",
    "                break\n",
    "\n",
    "            await sock.send(data.upper())\n",
    "\n",
    "    async def serve_forever(self):\n",
    "        while True:\n",
    "            sock, (addr, port) = await self.listen_sock.accept()\n",
    "            print('client connected addr={} port={}'.format(addr, port))\n",
    "\n",
    "            self.loop.add_coroutine(self.serve_client(sock))\n",
    "\n",
    "def main():\n",
    "    loop = EventLoop()\n",
    "    server = TcpServer(loop=loop)\n",
    "    loop.run_forever()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "speaking-recording",
   "metadata": {
    "hidden": true
   },
   "source": [
    "这个程序是一个精简的协程库实现，除了用于演示的应用代码 TcpServer ，整个库也就 100 来行代码！\n",
    "\n",
    "我们模仿常见协程库，引入 Future ，代表一个在未来才能获取到的数据。Future 一般由协程创建，典型的场景是这样的：协程在等待一个 IO 事件，这时它便创建一个 Future 对象，并把执行权归还给事件循环。\n",
    "\n",
    "例子中的 Future 类，有 4 个重要的属性：\n",
    "\n",
    "loop ，当前事件循环对象；\n",
    "done ，标识目标数据是否就绪；\n",
    "result ，目标数据；\n",
    "co ，关联协程，Future 就绪后，事件循环 loop 将把它放入可执行队列重新调度；\n",
    "注意到，Future 是一个 可等待对象 ( awaitable )，它实现了 __await__ 方法。当数据未就绪时，通过 yield 让出执行权，这时事件循环将协程记录在 Future 中。当数据就绪后，事件循环将协程放回可执行队列重新调度。\n",
    "\n",
    "协程库还将套接字进行 异步化 封装，抽象出 AsyncSocket 类，接口与原生 socket 对象类似。除了保存原生 socket 对象，它还保存事件循环对象，以便通过事件循环订阅 IO 事件。\n",
    "\n",
    "create_future_for_events 方法创建一个 Future 对象，来等待一个不知何时发生的 IO 事件。创建完 Future 对象后，进一步调用 loop 相关方法，将感兴趣的 IO 事件注册到 epoll 。当相关事件就绪时，事件循环将执行回调函数 handler ，它解除 epoll 注册，并将活跃事件作为目标数据设置到 Future 上 (注意 set_result 将唤醒协程)。\n",
    "\n",
    "然后是套接字系列操作函数，以 accept 为例，它不断尝试调用原生套接字，而原生套接字已被设为非阻塞。如果套接字已就绪，accept 将直接返回新连接，协程无须等待。\n",
    "\n",
    "否则，accept 方法抛出 BlockingIOError 异常。这时，协程调用 create_future_for_events 方法创建一个 Future 订阅读事件 ( EPOLLIN )，并等待事件到达。\n",
    "\n",
    "recv 、send 方法封装也是类似的，不同的是 send 需要订阅 可写事件 ( EPOLLOUT )。\n",
    "\n",
    "好了，终于来到协程库了主角事件循环 EventLoop 对象了，它有 3 个重要属性：\n",
    "\n",
    "epoll ，这是一个 epoll 描述符，用于订阅 IO 事件；\n",
    "runnables ，可执行协程队列；\n",
    "handlers ，IO 事件回调处理函数映射表；\n",
    "register_for_polling 方法注册感兴趣的 IO 事件和处理函数，它以文件描述符为键，将处理函数记录到映射表中，然后调用 epoll 完成事件订阅。unregister_from_polling 方法则刚好相反，用于取消注册。\n",
    "\n",
    "add_coroutine 将一个可运行的协程加入队列。run_coroutine 则调度一个可执行协程，它调用 send 将执行权交给协程。如果协程执行完毕，它将输出提示；协程需要等待时，会通过 yield 归还执行权并提交 Future 对象，它将协程记录到 Future 上下文。schedule_runnable_coroutines 将可执行协程逐个取出并调度，直到队列为空。\n",
    "\n",
    "run_forever 是事件循环的主体逻辑，这是一个永久循环。每次循环时，先调度可执行协程；然后通过 poll 等待协程注册的 IO 事件；当有新事件到达时，取出回调函数 handler 函数并调用。\n",
    "\n",
    "TcpServer 只是一个普通的协程式应用，无须赘述。接下来，我们逐步分析，看看程序启动后都发生什么事情：\n",
    "\n",
    "创建事件循环 EventLoop 对象，它将创建 epoll 描述符；\n",
    "创建 TcpServer 对象，它通过事件循环 loop 创建监听套接字，并将 serve_forever 协程放入可执行队列；\n",
    "事件循环 loop.run_forever 开始执行，它先调度可执行队列；\n",
    "可执行队列一开始只有一个协程 TcpServer.serve_forever ，它将开始执行 (由 run_coroutine 驱动)；\n",
    "执行权来到 TcpServer.serve_forever 协程，它调用 AsyncSocket.accept 准备接受一个新连接；\n",
    "假设原生套接字未就绪，它将抛出 BlockingIOError 异常；\n",
    "由于 IO 未就绪，协程创建一个 Future 对象，用来等待一个未来的 IO 事件 ( AsyncSocket.accept )；\n",
    "于此同时，协程调用事件循环 register_for_polling 方法订阅 IO 事件，并注册回调处理函数 handler ；\n",
    "future 是可以个可等待对象，await future 将执行权交给它的 __await__ 函数；\n",
    "由于一开始 future 是未就绪的，这时 yield 将协程执行逐层归还给事件循环，future 对象也被同时上报；\n",
    "执行权回到事件循环，run_coroutine 收到协程上报的 future 后将协程设置进去，以便 future 就绪后重新调度协程；\n",
    "可执行队列变空后，事件循环开始调用 epoll.poll 等待协程注册的 IO 事件 ( serve_forever )；\n",
    "当注册事件到达后，事件循环取出回调处理函数并调用；\n",
    "handler 先将套接字从 epoll 解除注册，然后调用 set_result 将活跃事件作为目标数据记录到 future 中；\n",
    "set_result 将协程重新放回可执行队列；\n",
    "IO 事件处理完毕，进入下一次事件循环；\n",
    "事件循环再次调度可执行队列，这时 TcpServer.serve_forever 协程再次拿到执行权；\n",
    "TcpServer.serve_forever 协程从 yield 语句恢复执行，开始返回目标数据，也就是先前设置的活跃事件；\n",
    "AsyncSocket.accept 内 await future 语句取得活跃事件，然后循环继续；\n",
    "循环再次调用原生套接字，这时它早已就绪，得到一个新套接字，简单包装后作为结果返回给调用者；\n",
    "TcpServer.serve_forever 拿到代表新连接的套接字后，创建一个 serve_client 协程并交给事件循环 loop ；\n",
    "TcpServer.serve_forever 进入下一次循环，调用 accept 准备接受下一个客户端连接；\n",
    "如果监听套接字未就绪，执行权再次回到事件循环；\n",
    "事件循环接着调度可执行队列里面的协程，*TcpServer.*serve_client 协程也开始执行了；\n",
    "etc\n",
    "这看着就像一个精密的机械装置，有条不紊的运行着，环环相扣！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "found-signal",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 工程实践：用 asyncio 协程构建高并发应用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vanilla-cambodia",
   "metadata": {
    "hidden": true
   },
   "source": [
    "互联网的基础是网络通讯，应用对网络请求的处理能力非常关键。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "useful-dover",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### C10K问题"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "orange-sister",
   "metadata": {
    "hidden": true
   },
   "source": [
    "在互联网尚未普及的早期，一台服务器同时在线100 个用户已经算是非常大型的应用了，工程上没有什么挑战。\n",
    "\n",
    "随着 Web 2.0 时代的到来，用户群体承几何倍数增长，服务器需要更强的并发处理能力才能承载海量的用户。这时，著名的 C10K 问题诞生了——如何让单台服务器同时支撑 1 万个客户端连接？\n",
    "\n",
    "最初的服务器应用编程模型，是基于进程/线程的：当一个新的客户端连接上来，服务器就分配一个进程或线程，来处理这个新连接。这意味着，想要解决 C10K 问题，操作系统需要同时运行 1 万个进程或线程。\n",
    "\n",
    "进程和线程是操作系统中，开销最大的资源之一。每个新连接都新开进程/线程，将造成极大的资源浪费。况且，受硬件资源制约，系统同一时间能运行的进程/线程数存在上限。\n",
    "\n",
    "换句话讲，在进程/线程模型中，每台服务器能处理的客户端连接数是非常有限的。为支持海量的业务，只能通过堆服务器这种简单粗暴的方式来实现。但这样的人海战术，既不稳定，也不经济。\n",
    "\n",
    "为了在单个进程/线程中同时处理多个网络连接，select 、 poll 、epoll 等 IO多路复用 技术应运而生。在 IO 多路复用模型，进程/线程不再阻塞在某个连接上，而是同时监控多个连接，只处理那些有新数据达到的活跃连接。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moderate-memorial",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 为什么需要协程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wooden-asian",
   "metadata": {
    "hidden": true
   },
   "source": [
    "单纯的 IO 多路复用编程模型，不像阻塞式编程模型那样直观，这为工程项目带来诸多不便。最典型的像 JavaScript 中的回调式编程模型，程序中各种 callback 函数满天飞，这不是一种直观的思维方式。\n",
    "\n",
    "为实现阻塞式那样直观的编程模型，协程(用户态线程)的概念被提出来。协程在进程/线程基础之上，实现多个执行上下文。由 epoll 等IO多路复用技术实现的事件循环，则负责驱动协程的调度、执行。\n",
    "\n",
    "协程可以看做是IO多路复用技术更高层次的封装。虽然与原始IO多路复用相比有一定的性能开销，但与进程/线程模型相比却非常突出。协程占用资源比进程/线程少，而且切换成本比较低。因此，协程在高并发应用领域潜力无限。\n",
    "\n",
    "然而，协程独特的运行机制，让初学者吃了不少亏，错漏百出。\n",
    "\n",
    "接下来，我们通过若干简单例子，探索协程应用之道，从中体会协程的作用，并揭示高并发应用设计、部署中存在的常见误区。由于 asyncio 是 Python 协程发展的主要趋势，例子便以 asyncio 为讲解对象。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "married-nashville",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 第一个协程应用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spectacular-adventure",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程应用由事件循环驱动，套接字必须是非阻塞模式，否则会阻塞事件循环。因此，一旦使用协程，就要跟很多类库说拜拜。以 MySQL 数据库操作为例，如果我们使用 asyncio ，就要用 aiomysql 包来连数据库。\n",
    "\n",
    "而想要开发 Web 应用，则可以用 aiohttp 包，它可以通过 pip 命令安装：\n",
    "\n",
    "这个例子实现一个完整 Web 服务器，虽然它只有返回当前时间的功能："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "skilled-prompt",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当一个新的请求到达时，aiohttp 将创建一个新协程来处理该请求，它将负责执行对应的处理函数。因此，处理函数必须是合法的协程函数，以 async 关键字开头。\n",
    "\n",
    "将程序跑起来后，我们就可以通过它获悉当前时间。在命令行中，可以用 curl 命令来发起请求：\n",
    "\n",
    "    from aiohttp import web\n",
    "    from datetime import datetime\n",
    "\n",
    "    async def handle(request):\n",
    "        return web.Response(text=datetime.now().strftime('%Y-%m-%d %H:%M:%S'))\n",
    "\n",
    "    app = web.Application()\n",
    "    app.add_routes([\n",
    "        web.get('/', handle),\n",
    "    ])\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        web.run_app(app)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "federal-visitor",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 压力测试"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expected-neutral",
   "metadata": {
    "hidden": true
   },
   "source": [
    "研发高并发应用，需要评估应用的处理能力。我们可以在短时间内发起大量的请求，并测算应用的吞吐能力。然而，就算你手再快，一秒钟也只能发起若干个请求呀。怎么办呢？\n",
    "\n",
    "我们需要借助一些压力测试工具，例如 Apache 工具集中的 ab 。如何安装使用 ab 不在本文的讨论范围，请参考这篇文章： Web压力测试 。\n",
    "\n",
    "事不宜迟，我们先以 100 为并发数，压 10000 个请求看看结果：\n",
    "\n",
    "- -n 选项，指定总请求数，即总共发多少个请求；\n",
    "- -c 选项，指定并发数，即同时发多少个请求；\n",
    "\n",
    "    $ ab -n 10000 -c 100 http://127.0.0.1:8080/\n",
    "    This is ApacheBench, Version 2.3 <$Revision: 1706008 $>\n",
    "    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/\n",
    "    Licensed to The Apache Software Foundation, http://www.apache.org/\n",
    "\n",
    "    Benchmarking 127.0.0.1 (be patient)\n",
    "    Completed 1000 requests\n",
    "    Completed 2000 requests\n",
    "    Completed 3000 requests\n",
    "    Completed 4000 requests\n",
    "    Completed 5000 requests\n",
    "    Completed 6000 requests\n",
    "    Completed 7000 requests\n",
    "    Completed 8000 requests\n",
    "    Completed 9000 requests\n",
    "    Completed 10000 requests\n",
    "    Finished 10000 requests\n",
    "\n",
    "\n",
    "    Server Software:        Python/3.8\n",
    "    Server Hostname:        127.0.0.1\n",
    "    Server Port:            8080\n",
    "\n",
    "    Document Path:          /\n",
    "    Document Length:        19 bytes\n",
    "\n",
    "    Concurrency Level:      100\n",
    "    Time taken for tests:   5.972 seconds\n",
    "    Complete requests:      10000\n",
    "    Failed requests:        0\n",
    "    Total transferred:      1700000 bytes\n",
    "    HTML transferred:       190000 bytes\n",
    "    Requests per second:    1674.43 [#/sec] (mean)\n",
    "    Time per request:       59.722 [ms] (mean)\n",
    "    Time per request:       0.597 [ms] (mean, across all concurrent requests)\n",
    "    Transfer rate:          277.98 [Kbytes/sec] received\n",
    "\n",
    "    Connection Times (ms)\n",
    "                  min  mean[+/-sd] median   max\n",
    "    Connect:        0    2   1.5      1      15\n",
    "    Processing:    43   58   5.0     57      89\n",
    "    Waiting:       29   47   6.3     47      85\n",
    "    Total:         43   60   4.8     58      90\n",
    "\n",
    "    Percentage of the requests served within a certain time (ms)\n",
    "      50%     58\n",
    "      66%     59\n",
    "      75%     60\n",
    "      80%     61\n",
    "      90%     65\n",
    "      95%     69\n",
    "      98%     72\n",
    "      99%     85\n",
    "     100%     90 (longest request)\n",
    "     \n",
    "从 ab 输出的报告中可以获悉，10000 个请求全部成功，总共耗时 5.972 秒，处理速度可以达到 1674.43 个每秒。\n",
    "\n",
    "现在，我们尝试提高并发数，看处理速度有没有提升：\n",
    "\n",
    "    $ ab -n 10000 -c 100 http://127.0.0.1:8080/\n",
    "    \n",
    "在 1000 并发数下，10000 个请求在 5.771 秒内完成，处理速度是 1732.87 ，略有提升但很不明显。这一点也不意外，例子中的处理逻辑绝大部分都是计算型，虚增并发数几乎没有任何意义。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manufactured-notion",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 协程擅长做什么"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "contained-trustee",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程擅长处理 IO 型的应用逻辑，举个例子，当某个协程在等待数据库响应时，事件循环将唤醒另一个就绪协程来执行，以此提高吞吐。为降低复杂性，我们通过在程序中睡眠来模拟等待数据库的效果。\n",
    "\n",
    "    import asyncio\n",
    "\n",
    "    from aiohttp import web\n",
    "    from datetime import datetime\n",
    "\n",
    "    async def handle(request):\n",
    "        # 睡眠一秒钟\n",
    "        asyncio.sleep(1)\n",
    "        return web.Response(text=datetime.now().strftime('%Y-%m-%d %H:%M:%S'))\n",
    "\n",
    "    app = web.Application()\n",
    "    app.add_routes([\n",
    "        web.get('/', handle),\n",
    "    ])\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        web.run_app(app)\n",
    "        \n",
    "|并发数\t|请求总数\t|耗时(秒)|\t处理速度(请求/秒)|\n",
    "|:-:|:-:|:-:|:-:|\n",
    "|100\t|10000\t|102.310|\t97.74|\n",
    "|500\t|10000\t|22.129|\t451.89|\n",
    "|1000|\t10000\t|12.780|\t782.50|\n",
    "\n",
    "可以看到，随着并发数的增加，处理速度也有明显的提升，趋势接近线性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "paperback-sodium",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 操作系统因素"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stone-elizabeth",
   "metadata": {
    "hidden": true
   },
   "source": [
    "当我们以 10000 并发对程序进行压测时，程序不断抛出 OSError 异常：\n",
    "\n",
    "    socket.accept() out of system resource\n",
    "    socket: <asyncio.TransportSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 8080)>\n",
    "    Traceback (most recent call last):\n",
    "      File \"/home/fasion/opt/python3.8.5/lib/python3.8/asyncio/selector_events.py\", line 162, in _accept_connection\n",
    "      File \"/home/fasion/opt/python3.8.5/lib/python3.8/socket.py\", line 292, in accept\n",
    "    OSError: [Errno 24] Too many open files\n",
    "    \n",
    "Linux 进程默认可以打开的文件描述符只有 1024 个：\n",
    "\n",
    "    $ ulimit -n\n",
    "    1024\n",
    "    \n",
    "当一个新请求连接达到时，事件循环将调用 accept 系统调用来接收。accept 系统调用将为新连接分配一个套接字( socket )，这将占用一个文件描述符。\n",
    "\n",
    "由于 10000 并发数远远超过 1024 ，程序的文件描述符很快就会被耗尽，这就是它抛异常的根源。要解决程序报错问题，唯一的途径就是将进程文件描述符数调大，用 ulimit 命令即可搞定：\n",
    "\n",
    "    $ ulimit -n 20000\n",
    "    $ ulimit -n\n",
    "    20000\n",
    "\n",
    "进程 **资源限制** ( rlimit )与 ulimit 命令的用法超出本文的范围，更多介绍请参考这篇文章：资源限制 。\n",
    "\n",
    "进程可打开的文件描述符数，直接制约进程承载的连接数，不可小视。网络应用部署时，如果忘记检查 rlimit 配置，很有可能造成故障。实际上，这类血案并不少见，我早年间也踩过坑。\n",
    "\n",
    "除了 rlimit ，内核参数，特别是与套接字相关的，在某些场景下也是需要特别注意的。因受篇幅所限，这里就不展开讨论了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "massive-norman",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 阻塞陷阱"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expected-location",
   "metadata": {
    "hidden": true
   },
   "source": [
    "协程由事件循环驱动，不能调用阻塞函数，不然将阻塞事件循环。事件循环一旦被阻塞，它便无法调度其他协程，只能等待阻塞函数返回。不信我们将 sleep 函数换成 time 模块中的阻塞版本，看看会发生什么事情？\n",
    "\n",
    "    import time\n",
    "\n",
    "    from aiohttp import web\n",
    "    from datetime import datetime\n",
    "\n",
    "    async def handle(request):\n",
    "        # 睡眠一秒钟\n",
    "        time.sleep(1)\n",
    "        return web.Response(text=datetime.now().strftime('%Y-%m-%d %H:%M:%S'))\n",
    "\n",
    "    app = web.Application()\n",
    "    app.add_routes([\n",
    "        web.get('/', handle),\n",
    "    ])\n",
    "\n",
    "    if __name__ == '__main__':\n",
    "        web.run_app(app)\n",
    "\n",
    "由于我们有预感程序的吞吐将大受影响，压测总请求数降为 100 ，并发数也降为从 1 起步：\n",
    "\n",
    "    $ ab -n 100 -c 1 http://127.0.0.1:8080/\n",
    "    \n",
    "测试结果如下，不管并发数是 1 、 10 还是 100 ，压测耗时都是 100 秒左右，处理速度都是 1 个每秒\n",
    "\n",
    "|并发数|\t请求总数|\t耗时(秒)|\t处理速度(请求/秒)|\n",
    "|:-:|:-:|:-:|:-:|\n",
    "|1\t|100|\t100.344|\t1.00|\n",
    "|10\t|100|\t102.248|\t0.98|\n",
    "|100|\t100|\t100.230|\t1.00|\n",
    "\n",
    "测试结果表明，一旦事件循环被阻塞，程序的并发处理能力将急剧下降。因此，千万不能在协程中调用同步阻塞的类库！\n",
    "\n",
    "在目前的 Python 的生态，大部分库仍然是同步的。比如 requests ，这是一个很好用的 http 请求库。但很不幸，它是同步的，会阻塞事件循环。因此，协程应用不能使用 requests ，只能使用 requests-async 或 aiohttp 中的替代品。\n",
    "\n",
    "那么，编写基于 asyncio 的协程应用，如何选择合适的类库呢？下面列举一些常用的替代品：\n",
    "\n",
    "- aiohttp ，发起 http 请求或实现 http 服务器；\n",
    "- requests-async ，异步版 requests ，用于发起 http 请求；\n",
    "- aiomysql ，用于连接 MySQL 数据库；\n",
    "- aiokafka ，用于连接 Kafka 消息队列；\n",
    "- aio_pika ，用于连接基于 AMQP 协议的消息队列，例如 RabbitMQ ；\n",
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "animal-nomination",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wanted-incidence",
   "metadata": {
    "hidden": true
   },
   "source": [
    "经过本节的实战练习，我们对协程的运行机制有了更深刻的认识，要点总结如下：\n",
    "\n",
    "- 协程擅长 IO密集型 场景，对 计算密集型 场景几乎没有任何意义；\n",
    "- 协程中不能调用同步阻塞式类库，否则将阻塞事件循环，性能骤降；\n",
    "- 进程可打开文件数、内存参数等操作系统因素，也会制约程序的处理能力，必须重视；\n",
    "\n",
    "协程有自己擅长的领域，也存在一些局限性。运用协程时，还需要注意避免一些误区。不过只要真正理解这些要点，相信协程式应用程序开发，是可以轻松驾驭的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interpreted-madrid",
   "metadata": {},
   "source": [
    "# 内存管理机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dried-landscape",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 小块内存池如何优化内存分配效率？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "australian-preliminary",
   "metadata": {
    "hidden": true
   },
   "source": [
    "内存是软件系统必不可少的物理资源，精湛的内存管理技术是确保内存使用效率的关键，也是进阶高级研发的必备技巧。为提高内存分配效率，Python 内部做了很多殚心竭虑的优化，从中我们可以获得一些启发。\n",
    "\n",
    "开始研究 Python 内存池之前，我们先大致了解下 Python 内存管理层次：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_01.jpg\" width=800 height=600> \n",
    "众所周知，计算机硬件资源由操作系统负责管理，内存资源也不例外。应用程序通过 系统调用 向操作系统申请内存，而 C 库函数则进一步将系统调用封装成通用的 **内存分配器** ，并提供了 malloc 系列函数。\n",
    "\n",
    "C 库函数实现的通用目的内存管理器是一个重要的分水岭，即内存管理层次中的 **第 0 层** 。此层之上是应用程序自己的内存管理，此层之下则是隐藏在冰山下方的操作系统部分。\n",
    "\n",
    "操作系统内部是一个基于页表的虚拟内存管理器 (**第-1层**)，以 页 ( page ) 为单位管理内存，CPU **内存管理单元** ( MMU ) 在这个过程中发挥重要作用。虚拟内存管理器下方则是底层存储设备 ( **第-2 层**)，直接管理物理内存以及磁盘等二级存储设备。\n",
    "\n",
    "绿色部分则是 Python 自己的内存管理，分为 3 层：\n",
    "\n",
    "- 第 1 层，是一个内存分配器，接管一切内存分配，内部是本文的主角 —— 内存池 ；\n",
    "- 第 2 层，在第 1 层提供的统一 PyMem_XXXX 接口基础上，实现统一的对象内存分配 ( object.tp_alloc )；\n",
    "- 第 3 层，为特定对象服务，例如前面章节介绍的 float 空闲对象缓存池；\n",
    "那么，Python 为什么不直接使用 malloc 系列函数，而是自己折腾一遍呢？原因主要是以下几点：\n",
    "\n",
    "- 引入内存池，可化解对象频繁创建销毁带来的内存分配压力；\n",
    "- 最大程度避免内存碎片化，提升内存利用效率；\n",
    "- malloc 有很多实现版本，不同实现性能千差万别；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "artificial-vector",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 内存碎片的挑战"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rising-storm",
   "metadata": {
    "hidden": true
   },
   "source": [
    "内存碎片化 是困扰经典内存分配器的一大难题，碎片化导致的结果也是惨重的。这是一个典型的内存碎片化例子：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_02.jpg\" width=800 height=600> \n",
    "\n",
    "虽然还有 1900K 的空闲内存，但都分散在一系列不连续的碎片上，甚至无法成功分配出 1000K 。\n",
    "\n",
    "那么，如何避免内存碎片化呢？想要解决问题，必先分析导致问题的根源。\n",
    "\n",
    "我们知道，应用程序请求内存块尺寸是不确定的，有大有小；释放内存的时机也是不确定的，有先有后。经典内存分配器将不同尺寸内存块混合管理，按照先来后到的顺序分配：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_03.jpg\" width=800 height=600> \n",
    "\n",
    "当大块内存回收后，可以被分为更小的块，然后分配出去：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_04.jpg\" width=800 height=600> \n",
    "\n",
    "而先分配的内存块未必先释放，慢慢地空洞就出现了：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_05.jpg\" width=800 height=600> \n",
    "\n",
    "随着时间的推移，碎片化会越来越严重，最终变得支离破碎：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_06.jpg\" width=800 height=600> \n",
    "\n",
    "由此可见，将不同尺寸内存块混合管理，将大块内存切分后再次分配的做法是罪魁祸首。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-brisbane",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 按尺寸分类管理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manual-juice",
   "metadata": {
    "hidden": true
   },
   "source": [
    "揪出内存碎片根源后，解决方案也就浮出水面了 —— 根据内存块尺寸，将内存空间划分成不同区域，独立管理。举个最简单的例子：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_07.jpg\" width=800 height=600> \n",
    "\n",
    "如图，内存被划分成小、中、大三个不同尺寸的区域，区域可由若干内存页组成，每个页都划分为统一规格的内存块。这样一来，小块内存的分配，不会影响大块内存区域，使其碎片化。\n",
    "\n",
    "每个区域的碎片仍无法完全避免，但这些碎片都是可以被重新分配出去的，影响不大。此外，通过优化分配策略，碎片还可被进一步合并。以小块内存为例，新内存优先从内存页 1 分配，内存页 2 将慢慢变空，最终将被整体回收。\n",
    "\n",
    "在 Python 虚拟机内部，时刻有对象创建、销毁，这引发频繁的内存申请、释放动作。这类内存尺寸一般不大，但分配、释放频率非常高，因此 Python 专门设计 内存池 对此进行优化。\n",
    "\n",
    "那么，尺寸多大的内存才会动用内存池呢？Python 以 512 字节为限，小于 512 的内存分配才会被内存池接管：\n",
    "\n",
    "- 0 ，直接调用 malloc 函数；\n",
    "- 1 ~ 512 ，由专门的内存池负责分配，内存池以内存尺寸进行划分；\n",
    "- 512 以上，直接调动 malloc 函数；\n",
    "\n",
    "那么，Python 是否为每个尺寸的内存都准备一个独立内存池呢？答案是否定的，原因有几个：\n",
    "\n",
    "- 内存规格有 512 种之多，如果内存池分也分 512 种，徒增复杂性；\n",
    "- 内存池种类越多，额外开销越大；\n",
    "- 如果某个尺寸内存只申请一次，将浪费内存页内其他空闲内存；\n",
    "\n",
    "相反，Python 以 8 字节为梯度，将内存块分为：8 字节、16 字节、24 字节，以此类推。总共 64 种：\n",
    "\n",
    "|请求大小|分配内存块大小|类别编号|\n",
    "|:--:|:--:|:--:|\n",
    "|1-8|8|0|\n",
    "|9-16|16|1|\n",
    "|17-24|24|2|\n",
    "|25-32|32|3|\n",
    "|…|…|…|\n",
    "|497-504|504|62|\n",
    "|505-512|512|63|\n",
    "\n",
    "以 8 字节内存块为例，内存池由多个 内存页 ( page ，一般是 4K ) 构成，每个内存页划分为若干 8 字节内存块：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_08.jpg\" width=800 height=600> \n",
    "   \n",
    "上图表示一个内存页，每个小格表示 1 字节，8 个字节组成一个块 ( block )。灰色表示空闲内存块，蓝色表示已分配内存块，深蓝色表示应用内存请求大小。\n",
    "\n",
    "只要请求的内存大小不超过 8 字节，Python 都在这个内存池为其分配一块 8 字节内存，就算只申请 1 字节内存也是如此。\n",
    "\n",
    "这种做法好处显而易见，前面提到的问题均得到解决，还带来另一个好处：内存起始地址均以计算机字为单位对齐。计算机以 字 ( word ) 为单位访问内存，因此内存以字对齐可提升内存读写速度。字大小从早期硬件的 2 字节、4 字节，慢慢发展到现在的 8 字节，甚至 16 字节。\n",
    "\n",
    "当然了，有得必有失，内存利用率成了被牺牲的因素，平均利用率为 (1+8)/2/8*100% ，大约只有 56.25% 。\n",
    "\n",
    "乍然一看，内存利用率有些惨不忍睹，但这只是 8 字节内存块的平均利用率。如果考虑所有内存块的平均利用率，其实数值并不低 —— 可以达到 98.65% 呢！计算方法如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "practical-therapy",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T09:47:29.329422Z",
     "start_time": "2021-03-30T09:47:29.319432Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98.65%\n"
     ]
    }
   ],
   "source": [
    "# 请求内存总量\n",
    "total_requested = 0\n",
    "# 实际分配内存总量\n",
    "total_allocated = 0\n",
    "\n",
    "# 请求内存从1到512字节\n",
    "for i in range(1, 513):\n",
    "    total_requested += i\n",
    "    # 实际分配内存为请求内存向上对齐为8的整数倍\n",
    "    total_allocated += (i+7)//8*8\n",
    "\n",
    "print('{:.2f}%'.format(total_requested/total_allocated*100))\n",
    "# 98.65%"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "present-diesel",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 内存池实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pleasant-knitting",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### pool"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "plastic-shannon",
   "metadata": {
    "hidden": true
   },
   "source": [
    "铺垫了这么多，终于可以开始研究源码，窥探 Python 内存池实现的秘密了，源码位于 Objects/obmalloc.c 。在源码中，我们发现对于 64 位系统，Python 将内存块大小定义为 16 字节的整数倍，而不是上述的 8 字节：\n",
    "\n",
    "    #if SIZEOF_VOID_P > 4\n",
    "    #define ALIGNMENT              16               /* must be 2^N */\n",
    "    #define ALIGNMENT_SHIFT         4\n",
    "    #else\n",
    "    #define ALIGNMENT               8               /* must be 2^N */\n",
    "    #define ALIGNMENT_SHIFT         3\n",
    "    #endif\n",
    "    \n",
    "为画图方便，我们仍然假设内存块为 8 字节的整数倍，即 (实际上，这些宏定义也是可配置的)：\n",
    "\n",
    "    #define ALIGNMENT               8\n",
    "    #define ALIGNMENT_SHIFT         3\n",
    "    \n",
    "Python 每次申请一个 内存页 ( page )，然后将其划分为统一尺寸的 内存块 ( block )，一个内存页大小是 4K ： \n",
    "\n",
    "    #define SYSTEM_PAGE_SIZE        (4 * 1024)\n",
    "    #define SYSTEM_PAGE_SIZE_MASK   (SYSTEM_PAGE_SIZE - 1)\n",
    "\n",
    "    #define POOL_SIZE               SYSTEM_PAGE_SIZE\n",
    "    #define POOL_SIZE_MASK          SYSTEM_PAGE_SIZE_MASK\n",
    "    \n",
    "Python 将内存页看做是由一个个内存块组成的池子 ( pool )，内存页开头是一个 pool_header 结构，用于组织当前页，并记录页中的空闲内存块：\n",
    "\n",
    "    /* Pool for small blocks. */\n",
    "    struct pool_header {\n",
    "        union { block *_padding;\n",
    "                uint count; } ref;          /* number of allocated blocks    */\n",
    "        block *freeblock;                   /* pool's free list head         */\n",
    "        struct pool_header *nextpool;       /* next pool of this size class  */\n",
    "        struct pool_header *prevpool;       /* previous pool       \"\"        */\n",
    "        uint arenaindex;                    /* index into arenas of base adr */\n",
    "        uint szidx;                         /* block size class index        */\n",
    "        uint nextoffset;                    /* bytes to virgin block         */\n",
    "        uint maxnextoffset;                 /* largest valid nextoffset      */\n",
    "    };\n",
    "    \n",
    "- count ，已分配出去的内存块个数；\n",
    "- freeblock ，指向空闲块链表的第一块；\n",
    "- nextpool ，用于将 pool 组织成链表的指针，指向下一个 pool ；\n",
    "- prevpool ，用于将 pool 组织成链表的指针，指向上一个 pool ；\n",
    "- szidx ，尺寸类别编号；\n",
    "- nextoffset ，下一个未初始化内存块的偏移量；\n",
    "- maxnextoffset ，合法内存块最大偏移量；\n",
    "\n",
    "当 Python 通过内存池申请内存时，如果没有可用 pool ，内存池将新申请一个 4K 页，并进行初始化。注意到，由于新内存页总是由内存请求触发，因此初始化时第一个内存块便已经被分配出去了：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_09.jpg\" width=800 height=600> \n",
    "随着内存分配请求的发起，空闲块将被分配出去。Python 将从灰色区域取出下一个作为空闲块，直到灰色块用光：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_10.jpg\" width=800 height=600> \n",
    "\n",
    "当有内存块被释放时，比如第一块，Python 将其链入空闲块链表头。请注意空闲块链表的组织方式 —— 每个块头部保存一个 next 指针，指向下一个空闲块：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_12.jpg\" width=800 height=600> \n",
    "\n",
    "这样一来，一个 pool 在其生命周期内，可能处于以下 3 种状态 (空闲内存块链表结构被省略，请自行脑补)：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_11.jpg\" width=800 height=600> \n",
    "\n",
    "- empty ，完全空闲 状态，内部所有内存块都是空闲的，没有任何块已被分配，因此 count 为 0 ；\n",
    "- used ，部分使用 状态，内部内存块部分已被分配，但还有另一部分是空闲的；\n",
    "- full ，完全用满 状态，内部所有内存块都已被分配，没有任何空闲块，因此 freeblock 为 NULL ；\n",
    "\n",
    "为什么要讨论 pool 状态呢？—— 因为 pool 的状态决定 Python 对它的处理策略：\n",
    "\n",
    "- 如果 pool 完全空闲，Python 可以将它占用的内存页归还给操作系统，或者缓存起来，后续需要分配新 pool 时直接拿来用；\n",
    "- 如果 pool 完全用满，Python 就无须关注它了，将它丢到一边；\n",
    "- 如果 pool 只是部分使用，说明它还有内存块未分配，Python 则将它们以 **双向循环链表** 的形式组织起来；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coordinated-showcase",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 可用 pool 链表"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "promotional-divide",
   "metadata": {
    "hidden": true
   },
   "source": [
    "由于 used 状态的 pool 只是部分使用，内部还有内存块未分配，将它们组织起来可供后续分配。Python 通过 pool_header 结构体中的 nextpool 和 prevpool 指针，将他们连成一个双向循环链表：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_13.jpg\" width=800 height=600> \n",
    "注意到，同个可用 pool 链表中的内存块大小规格都是一样的，上图以 16 字节类别为例。另外，为了简化链表处理逻辑，Python 引入了一个虚拟节点，这是一个常见的 C 语言链表实现技巧。一个空的 pool 链表是这样的，判断条件是 pool->nextpool == pool ：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_14.jpg\" width=800 height=600> \n",
    "虚拟节点只参与链表维护，并不实际管理内存块。因此，无须为虚拟节点分配一个完整的 4K 内存页，64 字节的 pool_header 结构体足矣。实际上，Python 作者们更抠，只分配刚好足够 nextpool 和 prevpool 指针用的内存，手法巧妙得令人瞠目结舌，我们稍后再表。\n",
    "\n",
    "Python 优先从链表第一个 pool 分配内存块，如果 pool 用满则将其从链表中剔除：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_15.jpg\" width=800 height=600> \n",
    "当一个内存块 ( block ) 被回收，Python 根据块地址计算得到 pool 地址。计算方法是大概是这样的：将 block 地址对齐为内存页 ( pool ) 尺寸的整数倍，便得到 pool 地址，具体请参看源码中的宏定义 POOL_ADDR 。\n",
    "\n",
    "得到 pool 地址后，Python 将空闲内存块插到空闲内存块链表头部。如果 pool 状态是由 完全用**满** ( full ) 变为 **可用** ( used )，Python 还会将它插回可用 pool 链表头部：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_16.jpg\" width=800 height=600> \n",
    "实际上，pool 链表任一节点均有机会完全空闲下来。这由概率决定，尾部节点概率最高，因此上图就这么画了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "meaningful-arthur",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### pool 链表数组"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "median-perry",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Python 内存池管理内存块，按照尺寸分门别类进行。因此，每种规格都需要维护一个独立的可用 pool 链表。如果以 8 字节为梯度，内存块规格可分 64 种之多 (见上表)。\n",
    "\n",
    "那么，如何组织这么多 pool 链表呢？最直接的方法是分配一个长度为 64 的虚拟节点数组：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_17.jpg\" width=800 height=600>\n",
    "   \n",
    "如果程序请求 5 字节，Python 将分配 8 字节内存块，通过数组第 0 个虚拟节点即可找到 8 字节 pool 链表；如果程序请求 56 字节，Python 将分配 64 字节内存块，则需要从数组第 7 个虚拟节点出发；其他以此类推。\n",
    "\n",
    "那么，虚拟节点数组需要占用多少内存呢？这不难计算：48 * 64 = 3072 = 3K48∗64=3072=3K\n",
    "\n",
    "哟，看上去还不少！Python 作者们可没这么大方，他们还从中抠出三分之二，具体是如何做到的呢？\n",
    "\n",
    "您可能已经注意到了，虚拟节点只参与维护链表结构，并不管理内存页。因此，虚拟节点其实只使用 pool_header 结构体中参与链表维护的 nextpool 和 prevpool 这两个指针字段：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_18.jpg\" width=800 height=600>\n",
    "   \n",
    "为避免浅蓝色部分内存浪费，Python 作者们将虚拟节点想象成一个个卡片，将深蓝色部分首尾相接，最终转换成一个纯指针数组。数组在 Objects/obmalloc.c 中定义，即 usedpools 。每个虚拟节点对应数组里面的两个指针：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_19.jpg\" width=800 height=600>\n",
    "接下来的一切交给想象力 —— 将两个指针前后的内存空间想象成自己的，这样就得到一个虚无缥缈的却非常完整的 pool_header 结构体（如下图左边虚线部分），我们甚至可以使用这个 pool_header 结构体的地址！由于我们不会访问除了 nextpool 和 prevpool 指针以外的字段，因此虽有内存越界，却也无伤大雅。\n",
    "\n",
    "下图以一个代表空链表的虚拟节点为例，nextpool 和 prevpool 指针均指向 pool_header 自己。虽然实际上 nextpool 和 prevpool 都指向了数组中的其他虚拟节点，但逻辑上可以想象成指向当前的 pool_header 结构体：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_20.jpg\" width=800 height=600>\n",
    "卧槽，这移花接木大法也太牛逼了吧！非常享受研究源码的过程，当年研究 Linux 内核数据结构中的链表实现时，也是大开眼界！\n",
    "\n",
    "经过这般优化，数组只需 16*64 = 1024 字节的内存空间即可，折合 1K ，节省了三分之二。为了节约这 2K 内存，代码变得难以理解。我第一次阅读源码时，在纸上花了半天才完全弄懂这个思路。\n",
    "\n",
    "效率与代码可读性有时是一对矛盾，如何选择见仁见智。不过，如果是日常项目，我多半不会为了 2K 内存而引入复杂性。Python 作为基础工具，能省则省。当然这个思路也有可能是在内存短缺的年代引入的，然后就这么一直用着。\n",
    "\n",
    "不管怎样，我还是决定将它写出来。如果你有兴趣研究 Objects/obmalloc.c 中的源码，就不用像我一样费劲，瞎耽误功夫。\n",
    "\n",
    "因篇幅关系，源码无法一一列举。对源码感兴趣的同学，请自己动手，丰衣足食。结合图示阅读，应该可以做到事半功倍。什么，不知道从何入手？—— 那就紧紧抓住这两个函数吧，一个负责分配，一个负责释放：\n",
    "\n",
    "- pymalloc_alloc\n",
    "- pymalloc_free\n",
    "\n",
    "虽然本节研究了很多东西，但还无法涵盖 Python 内存池的全部秘密，pool 的管理同样因篇幅关系无法展开。后续有机会我会接着写，感兴趣的童鞋请关注我。等不及？—— 源码欢迎您！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interesting-correspondence",
   "metadata": {},
   "source": [
    "## 引用计数的致命缺陷与应对之策"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "referenced-gibson",
   "metadata": {},
   "source": [
    "现代高级编程语言一般都内置垃圾回收机制，替程序员担起内存管理的重任，极大地提高了生产力。不同编程语言，采用的垃圾回收算法各有异同。那么，常见垃圾回收方法都有哪些呢？\n",
    "\n",
    "- 引用计数法 ( reference count )，对象记录引用次数，引用次数降为 0 时回收；\n",
    "- 标记-清除法 ( mark-sweep )，从根集合出发，遍历能访问到的对象并标记，将所有未标记的对象清除；\n",
    "- 复制法 ( copying )，将内存划分为大小相同的两块，一块用完后启用另一块并将存活的对象拷贝过去，原来那块则被整体回收；\n",
    "- 标记-整理法 ( mark-compact )，\n",
    "- 分代回收法 ( generational-collection )，根据存活时间将对象分为若干代(如新生代和老生代)，并按照不同代的特征采用最适合的回收策略；\n",
    "- etc\n",
    "\n",
    "引用计数可以说是最简单的垃圾回收方法，它能够在第一时间回收不再需要的对象，而且不会导致程序长时间停顿。由于引用计数存在一个致命缺陷，无情地限制了它的应用场景。\n",
    "\n",
    "我们在前面章节提过，Python 对象依靠引用计数机制来回收。每个 Python 对象都包含 ob_refcnt 字段，这个字段记录对象的引用次数。那么，Python 是如何解决引用计数的致命缺陷的呢？它到底用了什么黑科技？\n",
    "\n",
    "让我们带着这些疑问，开始本节关于引用计数法的学习。为全面研究 Python 垃圾回收机制，打下坚实的基础。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "miniature-jacksonville",
   "metadata": {},
   "source": [
    "### 引用计数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "associate-floor",
   "metadata": {},
   "source": [
    "引用计数 是计算机编程语言中的一种 内存管理技术 ，它将资源被引用的次数保存起来，当引用次数变为 0 时就将资源释放。它管理的资源并不局限于内存，还可以是对象、磁盘空间等等。\n",
    "\n",
    "Python 也使用引用计数这种方式来管理内存，每个 Python 对象都包含一个公共头部，头部中的 ob_refcnt 字段便用于维护对象被引用次数。回忆对象模型部分内容，我们知道一个典型的 Python 对象结构如下：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_21.jpg\" width=800 height=600> \n",
    "\n",
    "当创建一个对象实例时，先在堆上为对象申请内存，对象的引用计数被初始化为 1 。以 Python 为例，我们创建一个 float 对象保存圆周率，并把它赋值到变量 pi ：\n",
    "\n",
    "\n",
    "    \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "accepted-thesis",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:53:21.772436Z",
     "start_time": "2021-03-30T11:53:21.748436Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.14"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> pi = 3.14\n",
    ">>> pi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-force",
   "metadata": {},
   "source": [
    "由于此时只有变量 pi 引用 float 对象，因此它的引用计数为 1 ：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_22.jpg\" width=800 height=600> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "danish-identification",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:53:29.831433Z",
     "start_time": "2021-03-30T11:53:29.812433Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.14"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> f = pi\n",
    ">>> f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "other-track",
   "metadata": {},
   "source": [
    "当我们把 pi 赋值给 f 后，float 对象的引用计数就变成了 2 ，因为现在有两个变量引用它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "growing-spirit",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:53:36.674442Z",
     "start_time": "2021-03-30T11:53:36.662442Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3.14]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> l = [f]\n",
    ">>> l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "capable-plane",
   "metadata": {},
   "source": [
    "我们新建一个 list 对象，并把 float 对象保存在里面。这样一来，float 对象有多了一个来自 list 对象的引用，因此它的引用计数又加一，变成 3 了：\n",
    "\n",
    "标准库 sys 模块中有一个函数 getrefcount 可以获取对象引用计数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "thorough-milan",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:54:20.365444Z",
     "start_time": "2021-03-30T11:54:20.347446Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> import sys\n",
    ">>> sys.getrefcount(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "automated-method",
   "metadata": {},
   "source": [
    "咦！引用计数不应该是 3 吗？为什么会是 4 呢？由于 float 对象被作为参数传给 getrefcount 函数，它在函数执行过程中作为函数的局部变量存在，因此又多了一个引用：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_23.jpg\" width=800 height=600> \n",
    "   \n",
    "随着 getrefcount 函数执行完毕并返回，它的栈帧对象将从调用链中解开并销毁，这时 float 对象的引用计数也跟着下降。因此，当一个对象作为参数传个函数后，它的引用计数将加一；当函数返回，局部名字空间销毁后，对象引用计数又加一。理解这些后，getrefcount 的行为也就解释得通了。\n",
    "\n",
    "引用计数就这样随着引用关系的变动，不断变化着。当所有引用都消除后，引用计数就降为零，这时 Python 就可以安全地销毁对象，回收内存了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "removable-vegetation",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:56:12.434259Z",
     "start_time": "2021-03-30T11:56:12.418259Z"
    }
   },
   "outputs": [],
   "source": [
    ">>> del l\n",
    ">>> del f\n",
    ">>> del pi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "logical-oasis",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:56:20.797452Z",
     "start_time": "2021-03-30T11:56:20.639450Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pi' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mNameError\u001B[0m                                 Traceback (most recent call last)",
      "\u001B[1;32m<ipython-input-6-b8d3361a8e9d>\u001B[0m in \u001B[0;36m<module>\u001B[1;34m\u001B[0m\n\u001B[1;32m----> 1\u001B[1;33m \u001B[0msys\u001B[0m\u001B[1;33m.\u001B[0m\u001B[0mgetrefcount\u001B[0m\u001B[1;33m(\u001B[0m\u001B[0mpi\u001B[0m\u001B[1;33m)\u001B[0m\u001B[1;33m\u001B[0m\u001B[1;33m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[1;31mNameError\u001B[0m: name 'pi' is not defined"
     ]
    }
   ],
   "source": [
    ">>> sys.getrefcount(pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "soviet-monday",
   "metadata": {},
   "source": [
    "### 循环引用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "forward-pavilion",
   "metadata": {},
   "source": [
    "对象引用关系构成一张 有向图 ，这张图可以很复杂。但如果图中有环，形成 循环应用 关系，引用计数法的表现就比较微妙了。先考察这个简单的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "danish-european",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:57:51.411026Z",
     "start_time": "2021-03-30T11:57:51.401028Z"
    }
   },
   "outputs": [],
   "source": [
    "class Car:\n",
    "    \n",
    "    def __init__(self, factory):\n",
    "        self.factory = factory\n",
    "\n",
    "class CarFactory:\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.cars = []\n",
    "        \n",
    "    def build_car(self):\n",
    "        car = Car(factory=self)\n",
    "        self.cars.append(car)\n",
    "        return car"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manual-initial",
   "metadata": {},
   "source": [
    "这个例子定义了两个类，Car 表示一辆汽车，它有一个属性 factory 指向制造它的汽车工厂；CarFactory 表示一个汽车工厂，它有一个列表 cars ，保存它制造的每辆汽车。这样一来，Car 和 CarFactory 的实例对象构成了互相引用的关系。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "saving-repair",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T11:57:54.821372Z",
     "start_time": "2021-03-30T11:57:54.814369Z"
    }
   },
   "outputs": [],
   "source": [
    ">>> factory = CarFactory()\n",
    ">>> car1 = factory.build_car()\n",
    ">>> car2 = factory.build_car()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "industrial-wiring",
   "metadata": {},
   "source": [
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_23.jpg\" width=800 height=600> \n",
    "CarFactory 通过 cars 属性引用一个列表对象，列表对象又引用两个 Car 对象，而 Car 对象又通过 factory 属性引用 CarFactory 对象。因此，上面这个引用关系图中，形成了两个 环 。那么，环会导致什么问题呢？\n",
    "\n",
    "假设，我们将 factory 、car1 以及 car2 删掉，讲道理这 3 个对象应该回收，因为已经没有其他地方引用它们了。事实是不是这样的呢？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "unlike-brass",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-30T12:00:04.076451Z",
     "start_time": "2021-03-30T12:00:04.062456Z"
    }
   },
   "outputs": [],
   "source": [
    ">>> del factory\n",
    ">>> del car1\n",
    ">>> del car2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unexpected-middle",
   "metadata": {},
   "source": [
    "实际上，由于内部循环引用的存在，就算我们将外部变量删除，对象的引用计数也不为零，无法回收：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_26.jpg\" width=800 height=600> \n",
    "\n",
    "这个问题可就严重了，该回收的对象无法回收，不就内存泄露了吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "experienced-airplane",
   "metadata": {},
   "source": [
    "### 标记-清除"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "catholic-intelligence",
   "metadata": {},
   "source": [
    "那么，如何解决循环引用带来的问题呢？解决问题的思路主要有两个：\n",
    "\n",
    "- 精心设计程序，确保对象不会形成环状引用关系(被动避免)；\n",
    "- 实现算法定期检查识别需要回收的垃圾对象，进而将它们回收(主动回收)；\n",
    "\n",
    "被动避免的思路虽理论上可行，但基本没有可操作性。一方面，由于非循环引用这个约束存在，程序设计必然要牺牲一些灵活性；另一方面，由应用程序自己保证的做法并不可靠，稍有失误就会造成内存泄露，死无葬身之地。\n",
    "\n",
    "主动思路一般分为两步：**垃圾识别** 和 **垃圾回收** 。垃圾对象被识别出来后，回收就只是自然而然的工作了，因此垃圾识别是解决问题的关键。那么，有什么办法可以将垃圾对象识别出来呢？我们来考察一个一般化例子：\n",
    "\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_27.jpg\" width=800 height=600> \n",
    "   \n",
    "这是一个对象引用关系图，其中灰色部分是需要回收但由于循环引用而无法回收的垃圾对象，绿色部分是被程序引用而不能回收的活跃对象。如果我们能够将活跃对象逐个遍历并标记，那么最后没有被标记的对象就是垃圾对象。\n",
    "\n",
    "遍历活跃对象，第一步需要找出 根对象 ( root object )集合。所谓根对象，就是指被全局引用或者在栈中引用的对象，这部对象是不能被删除的。因此，我们将这部分对象标记为绿色，作为活跃对象遍历的起点。\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_28.jpg\" width=800 height=600> \n",
    "根对象本身是 可达的 ( reachable )，不能删除；被根对象引用的对象也是可达的，同样不能删除；以此类推。我们从一个根对象出发，沿着引用关系遍历，遍历到的所有对象都是可达的，不能删除。\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_29.jpg\" width=800 height=600> \n",
    "这样一来，当我们遍历完所有根对象，活跃对象也就全部找出来了：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_30.jpg\" width=800 height=600> \n",
    "而没有被标色的对象就是 不可达 ( unreachable )的垃圾对象，可以被安全回收。循环引用的致命缺陷完美解决了！\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_31.jpg\" width=800 height=600> \n",
    "这就是垃圾回收中常用的 标记清除法 ，思想其实平白无奇。\n",
    "\n",
    "Python 内部也采用这个办法来识别、回收垃圾对象，但细节略有差异。由于 Python 绝大部分对象都是在堆中分配的，根对象集合不太直观。因此，Python 先通过一个算法找出根对象，然后再从根对象出发遍历可达的活跃对象。此外，为提升垃圾回收效率，Python 还引入分代回收机制。当中来龙去脉，请听下回分解。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interim-attack",
   "metadata": {},
   "source": [
    "## 全面解读 Python 垃圾回收机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verified-person",
   "metadata": {},
   "source": [
    "Python 内部采用 引用计数法 ，为每个对象维护引用次数，并据此回收不再需要的垃圾对象。由于引用计数法存在重大缺陷，循环引用时有内存泄露风险，因此 Python 还采用 标记清除法 来回收存在循环引用的垃圾对象。此外，为了提高垃圾回收( GC )效率，Python 还引入了 分代回收机制 。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "endangered-gabriel",
   "metadata": {},
   "source": [
    "### 对象跟踪"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "twelve-workshop",
   "metadata": {},
   "source": [
    "将程序内部对象跟踪起来，是实现垃圾回收的第一步。那么，是不是程序创建的所有对象都需要跟踪呢？\n",
    "\n",
    "一个对象是否需要跟踪，取决于它会不会形成循环引用。按照引用特征，Python 对象可以分为两类：\n",
    "\n",
    "- 内向型对象 ，例如 int 、float 、 str 等，这类对象不会引用其他对象，因此无法形成循环引用，无须跟踪；\n",
    "- 外向型对象 ，例如 tuple 、 list 、 dict 等容器对象，以及函数、类实例等复杂对象，这类对象一般都会引用其他对象，存在形成循环引用的风险，因此是垃圾回收算法关注的重点；\n",
    "这是一个典型的例子，橘红色外向型对象存在循环引用的可能性，需要跟踪；而绿色内向型对象在引用关系图中只能作为叶子节点存在，无法形成任何环状，因此无需跟踪：\n",
    "   <img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_32.jpg\" width=800 height=600> \n",
    "Python 为外向型对象分配内存时，调用位于 Modules/gcmodule.c 源文件的 _PyObject_GC_Alloc 函数。该函数在对象头部之前预留了一些内存空间，以便垃圾回收模块用 **链表** 将它们跟踪起来。预留的内存空间是一个 _gc_head 结构体，它定义于 Include/objimpl.h 头文件：\n",
    "\n",
    "    typedef union _gc_head {\n",
    "        struct {\n",
    "            union _gc_head *gc_next;\n",
    "            union _gc_head *gc_prev;\n",
    "            Py_ssize_t gc_refs;\n",
    "        } gc;\n",
    "        long double dummy;  /* force worst-case alignment */\n",
    "        // malloc returns memory block aligned for any built-in types and\n",
    "        // long double is the largest standard C type.\n",
    "        // On amd64 linux, long double requires 16 byte alignment.\n",
    "        // See bpo-27987 for more discussion.\n",
    "    } PyGC_Head;\n",
    "    \n",
    "- gc_next ，链表后向指针，指向后一个被跟踪的对象；\n",
    "- gc_prev ，链表前向指针，指向前一个被跟踪的对象；\n",
    "- gc_refs ，对象引用计数副本，在标记清除算法中使用；\n",
    "- dummy ，内存对齐用，以 64 位系统为例，确保 _gc_head 结构体大小是 16 字节的整数倍，结构体地址以 16 字节为单位对齐；\n",
    "\n",
    "以 list 对象为例，_PyObject_GC_Alloc 函数在 PyListObject 结构体基础上加上 _gc_head 结构体来申请内存，但只返回 PyListObject 的地址作为对象地址，而不是整块内存的首地址：   \n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_33.jpg\" width=800 height=600> \n",
    "就这样，借助 gc_next 和 gc_prev 指针，Python 将需要跟踪的对象一个接一个组织成 双向链表 ：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_34.jpg\" width=800 height=600> \n",
    "这个链表也被称为 **可收集** ( collectable )对象链表，Python 将从这个链表中收集并回收垃圾对象。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wanted-details",
   "metadata": {},
   "source": [
    "### 分代回收机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "capital-willow",
   "metadata": {},
   "source": [
    "Python 程序启动后，内部可能会创建大量对象。如果每次执行标记清除法时，都需要遍历所有对象，多半会影响程序性能。为此，Python 引入分代回收机制——将对象分为若干“代”( generation )，每次只处理某个代中的对象，因此 GC 卡顿时间更短。\n",
    "\n",
    "那么，按什么标准划分对象呢？是否随机将一个对象划分到某个代即可呢？答案是否定的。实际上，对象分代里头也是有不少学问的，好的划分标准可显著提升垃圾回收的效率。\n",
    "\n",
    "考察对象的生命周期，可以发现一个显著特征：一个对象存活的时间越长，它下一刻被释放的概率就越低。我们应该也有这样的亲身体会：经常在程序中创建一些临时对象，用完即刻释放；而定义为全局变量的对象则极少释放。\n",
    "\n",
    "因此，根据对象存活时间，对它们进行划分就是一个不错的选择。对象存活时间越长，它们被释放的概率越低，可以适当降低回收频率；相反，对象存活时间越短，它们被释放的概率越高，可以适当提高回收频率。\n",
    "\n",
    "|对象存活时间|\t释放概率|\t回收频率|\n",
    "|:-:|:-:|:-:|\n",
    "|长|\t低|\t低|\n",
    "|短|\t高|\t高|\n",
    "\n",
    "Python 内部根据对象存活时间，将对象分为 3 代(见 Include/internal/mem.h )：\n",
    "\n",
    "    #define NUM_GENERATIONS 3\n",
    "    \n",
    "每个代都由一个 gc_generation 结构体来维护，它同样定义于 Include/internal/mem.h 头文件：\n",
    "\n",
    "    struct gc_generation {\n",
    "        PyGC_Head head;\n",
    "        int threshold; /* collection threshold */\n",
    "        int count; /* count of allocations or collections of younger\n",
    "                      generations */\n",
    "    };\n",
    "    \n",
    "- head ，可收集对象链表头部，代中的对象通过该链表维护；\n",
    "- threshold ，仅当count 超过本阀值时，Python 垃圾回收操作才会扫描本代对象；\n",
    "- count ，计数器，不同代统计项目不一样；\n",
    "\n",
    "Python 虚拟机运行时状态由 Include/internal/pystate.h 中的 pyruntimestate 结构体表示，它内部有一个 _gc_runtime_state ( Include/internal/mem.h )结构体，保存 GC 状态信息，包括 3 个对象代。这 3 个代，在 GC 模块( Modules/gcmodule.c ) _PyGC_Initialize 函数中初始化：\n",
    "\n",
    "    struct gc_generation generations[NUM_GENERATIONS] = {\n",
    "        /* PyGC_Head,                                 threshold,      count */\n",
    "        {{{_GEN_HEAD(0), _GEN_HEAD(0), 0}},           700,            0},\n",
    "        {{{_GEN_HEAD(1), _GEN_HEAD(1), 0}},           10,             0},\n",
    "        {{{_GEN_HEAD(2), _GEN_HEAD(2), 0}},           10,             0},\n",
    "    };\n",
    "    \n",
    "为方便讨论，我们将这 3 个代分别称为：初生代、中生代 以及 老生代。当这 3 个代初始化完毕后，对应的 gc_generation 数组大概是这样的：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_35.jpg\" width=800 height=600> \n",
    "每个 gc_generation 结构体链表头节点都指向自己，换句话说每个可收集对象链表一开始都是空的；计数器字段 count 都被初始化为 0 ；而阀值字段 threshold 则有各自的策略。这些策略如何理解呢？\n",
    "\n",
    "Python 调用 _PyObject_GC_Alloc 为需要跟踪的对象分配内存时，该函数将初生代 count 计数器加一，随后对象将接入初生代对象链表；当 Python 调用 PyObject_GC_Del 释放垃圾对象内存时，该函数将初生代 count 计数器减一；_PyObject_GC_Alloc 自增 count 后如果超过阀值( 700 )，将调用 collect_generations 执行一次垃圾回收( GC )。\n",
    "\n",
    "collect_generations 函数从老生代开始，逐个遍历每个生代，找出需要执行回收操作( count>threshold )的最老生代。随后调用 collect_with_callback 函数开始回收该生代，而该函数最终调用 collect 函数。\n",
    "\n",
    "collect 函数处理某个生代时，先将比它年轻的生代计数器 count 重置为 0 ；然后将它们的对象链表移除，与自己的拼接在一起后执行 GC 算法(本文后半部分介绍)；最后，将下一个生代计数器加一。\n",
    "\n",
    "- 系统每新增 701 个需要 GC 的对象，Python 就执行一次 GC 操作；\n",
    "- 每次 GC 操作需要处理的生代可能是不同的，由 count 和 threshold 共同决定；\n",
    "- 某个生代需要执行 GC ( count>threshold )，在它前面的所有年轻生代也同时执行 GC ；\n",
    "- 对多个代执行 GC ，Python 将它们的对象链表拼接在一起，一次性处理；\n",
    "- GC 执行完毕后，count 清零，而后一个生代 count 加一；\n",
    "\n",
    "下面是一个简单的例子：初生代触发 GC 操作，Python 执行 collect_generations 函数。它找出了达到阀值的最老生代是中生代，因此调用 collection_with_callback(1) ，1 是中生代在数组中的下标。\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_36.jpg\" width=800 height=600> \n",
    "collection_with_callback(1) 最终执调用 collect(1) ，它先将后一个生代计数器加一；然后将本生代以及前面所有年轻生代计数器重置为零；最后调用 gc_list_merge 将这几个可回收对象链表合并在一起：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_37.jpg\" width=800 height=600> \n",
    "最后，collect 函数执行标记清除算法，对合并后的链表进行垃圾回收，具体细节在本文后半部分展开介绍。\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_38.jpg\" width=800 height=600> \n",
    "这就是分代回收机制的全部秘密，它看似复杂，但只需略加总结就可以得到几条直白的策略：\n",
    "\n",
    "- 每新增 701 个需要 GC 的对象，触发一次新生代 GC ；\n",
    "- 每执行 11 次新生代 GC ，触发一次中生代 GC ；\n",
    "- 每执行 11 次中生代 GC ，触发一次老生代 GC (老生代 GC 还受其他策略影响，频率更低)；\n",
    "- 执行某个生代 GC 前，年轻生代对象链表也移入该代，一起 GC ；\n",
    "- 一个对象创建后，随着时间推移将被逐步移入老生代，回收频率逐渐降低；\n",
    "\n",
    "由于篇幅关系，分代回收部分代码无法逐行解释，请对照图示阅读相关重点函数，应该不难理解。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "detected-lawyer",
   "metadata": {},
   "source": [
    "### 标记清除法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "large-optics",
   "metadata": {},
   "source": [
    "现在，我们对 Python 垃圾回收机制有了框架上的把握，但对检测垃圾对象的方法还知之甚少。垃圾对象识别是垃圾回收工作的重中之重，Python 是如何解决这一关键问题的呢？\n",
    "\n",
    "前面已经提过，Python 采用标记清除法识别垃圾对象。该算法的输入是可收集对象链表，链表给出所有需要检测的对象；算法的输出是两个链表，其中一个包含 可达 ( reachable )对象，另一个包含 不可达 ( unreachable )对象。\n",
    "\n",
    "标记清除算法在 collect 函数中实现，它位于 Modules/gcmodule.c ，步骤并不复杂。为避免深陷源码细节，我们先抽出身来进行一次直观考察。假设待检测可收集对象链表中的对象引用关系如下：\n",
    "\n",
    "其中，数字表示引用计数，箭头表示引用关系，虚线表示链表外对象(来自其他年代)，而链表结构被省略了。\n",
    "\n",
    "首先，我们需要找出根对象。这里根对象是指被本链表以外的对象引用或被 Python 虚拟机直接引用的对象，与上一小节讨论的略有不同。由于根对象存在来自外部的引用，不能安全释放，应该标记为 可达 ( reachable )。\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_39.jpg\" width=800 height=600> \n",
    "根对象集合不难确定：我们只需遍历每个对象引用的对象，将它们的引用计数减一，最后计数不为零的就是根对象。\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_40.jpg\" width=800 height=600> \n",
    "请注意，虚线表示外部对象，它既不会被遍历到，引用计数也不会被减一。接着，从深色的根对象出发，遍历引用关系，将可以访问到的对象同样标为深色。标色完毕后，对象将分成有色和无色两种：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_41.jpg\" width=800 height=600> \n",
    "这样一来，无色部分就是只存在循环引用的垃圾对象，可以被安全释放。\n",
    "\n",
    "回过头来研究位于 collect 函数的算法处理逻辑，它先将对象引用计数拷贝到 gc_refs 字段：\n",
    "    update_refs(young);\n",
    "这是因为直接操作 ob_refcnt 字段的话，对象的引用计数就被破坏了，而且无法复原。操作 gc_refs 副本字段，就不存在这个问题。\n",
    "\n",
    "接着，collect 函数调用 subtract_refs 遍历链表中每个对象，将它们引用的对象引用计数( gc_refs )减一。注意到，subtract_refs 函数调用 tp_traverse 函数，来遍历被一个对象引用的对象：\n",
    "    subtract_refs(young);\n",
    "    \n",
    "    static int\n",
    "    visit_decref(PyObject *op, void *data)\n",
    "    {\n",
    "        assert(op != NULL);\n",
    "        if (PyObject_IS_GC(op)) {\n",
    "            PyGC_Head *gc = AS_GC(op);\n",
    "            assert(_PyGCHead_REFS(gc) != 0); /* else refcount was too small */\n",
    "            if (_PyGCHead_REFS(gc) > 0)\n",
    "                // 将对象引用计数减一(gc_refs)\n",
    "                _PyGCHead_DECREF(gc);\n",
    "        }\n",
    "        return 0;\n",
    "    }\n",
    "\n",
    "    static void\n",
    "    subtract_refs(PyGC_Head *containers)\n",
    "    {\n",
    "        traverseproc traverse;\n",
    "        PyGC_Head *gc = containers->gc.gc_next;\n",
    "        // 遍历链表每一个对象\n",
    "        for (; gc != containers; gc=gc->gc.gc_next) {\n",
    "            // 遍历当前对象所引用的对象，调用visit_decref将它们的引用计数减一\n",
    "            traverse = Py_TYPE(FROM_GC(gc))->tp_traverse;\n",
    "            (void) traverse(FROM_GC(gc),\n",
    "                           (visitproc)visit_decref,\n",
    "                           NULL);\n",
    "        }\n",
    "    }\n",
    "经过这个步骤之后，根对象就被找出来了，它们的引用计数( gc_refs )不为零。\n",
    "\n",
    "最后，collect 函数初始化一个链表 unreachable 来保存不可达对象，调用 move_unreachable 标记可达对象，并将不可达对象移入 unreachable 链表：\n",
    "\n",
    "    gc_list_init(&unreachable);\n",
    "    move_unreachable(young, &unreachable);\n",
    "    \n",
    "move_unreachable 遍历给定对象链表，如果一个对象的引用计数不为 0 ，就将它标记为可达( GC_REACHABLE )；否则将它标记为临时不可达，移入不可达链表。\n",
    "\n",
    "    // 遍历链表每个对象\n",
    "    while (gc != young) {\n",
    "        PyGC_Head *next;\n",
    "\n",
    "        // 如果引用计数不为0，说明它可达\n",
    "        if (_PyGCHead_REFS(gc)) {\n",
    "            PyObject *op = FROM_GC(gc);\n",
    "            traverseproc traverse = Py_TYPE(op)->tp_traverse;\n",
    "            assert(_PyGCHead_REFS(gc) > 0);\n",
    "            // 将它标记为可达\n",
    "            _PyGCHead_SET_REFS(gc, GC_REACHABLE);\n",
    "            // 遍历被它引用的对象，调用visit_reachable将被引用对象标记为可达\n",
    "            (void) traverse(op,\n",
    "                            (visitproc)visit_reachable,\n",
    "                            (void *)young);\n",
    "            next = gc->gc.gc_next;\n",
    "            if (PyTuple_CheckExact(op)) {\n",
    "                _PyTuple_MaybeUntrack(op);\n",
    "            }\n",
    "        }\n",
    "        else {\n",
    "            next = gc->gc.gc_next;\n",
    "            // 暂时移入不可达链表\n",
    "            gc_list_move(gc, unreachable);\n",
    "            _PyGCHead_SET_REFS(gc, GC_TENTATIVELY_UNREACHABLE);\n",
    "        }\n",
    "        gc = next;\n",
    "    }\n",
    "    \n",
    "如果一个对象可达，Python 还通过 tp_traverse 逐个遍历它引用的对象，并调用 visit_reachable 函数进行标记：\n",
    "\n",
    "    static int\n",
    "    visit_reachable(PyObject *op, PyGC_Head *reachable)\n",
    "    {\n",
    "        if (PyObject_IS_GC(op)) {\n",
    "            PyGC_Head *gc = AS_GC(op);\n",
    "            const Py_ssize_t gc_refs = _PyGCHead_REFS(gc);\n",
    "\n",
    "            if (gc_refs == 0) {\n",
    "                // 引用计数为零，将计数设置为1，表明它是可达的\n",
    "                _PyGCHead_SET_REFS(gc, 1);\n",
    "            }\n",
    "            else if (gc_refs == GC_TENTATIVELY_UNREACHABLE) {\n",
    "                // 如果对象被临时标记为不可达，将引用计数设置为1表明它是可达的\n",
    "                // 并移回原链表继续处理\n",
    "                gc_list_move(gc, reachable);\n",
    "                _PyGCHead_SET_REFS(gc, 1);\n",
    "            }\n",
    "             else {\n",
    "                assert(gc_refs > 0\n",
    "                       || gc_refs == GC_REACHABLE\n",
    "                       || gc_refs == GC_UNTRACKED);\n",
    "             }\n",
    "        }\n",
    "        return 0;\n",
    "    }\n",
    "    \n",
    "如果被引用的对象引用计数为 0 ，将它的引用计数设为 1 ，之后它将被 move_unreachable 遍历到并设为可达；如果被引用的对象被临时移入 unreachable 链表，同样将它的引用计数设为 1 ，并从 unreachable 链表移回原链表尾部，之后它将被 move_unreachable 遍历到并设为可达。\n",
    "\n",
    "当 move_unreachable 函数执行完毕，unreachable 链表中的对象就是不可达对象，可被安全回收。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hourly-breath",
   "metadata": {},
   "source": [
    "## 妙手回春：内存泄露诊断案例分析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "foster-friendship",
   "metadata": {},
   "source": [
    "虽然 Python 自带垃圾回收机制，替开发人员管理内存，并不意味着 Python 程序没有内存泄露之虞。实际上，Python 程序内存泄露问题时有发生 —— 程序跑着跑着，占用内存越来越多，最后只能动用重启大法释放内存……\n",
    "\n",
    "由于内存分配回收工作已被 Python 接管，内存泄露问题排查起来相对来说也比较晦涩。正常情况下，引用计数 机制确保对象没有引用时释放，而 标记清除 则解决了 循环引用 的问题，理论上不存在内存泄露的可能性。\n",
    "\n",
    "那么，Python 程序内存泄露问题一般是如何造成的呢？程序员的失误是其中的主要原因，最常见的是下面两点：\n",
    "\n",
    "- 容器泄露 ，使用容器对象存储数据，但数据只进不出，没有清理机制，容器便慢慢变大，最后撑爆内存；\n",
    "- __del__魔术方法误用，如果对象实现了 __del__ 魔术方法，Python 就无法用标记清除法解决循环引用问题，这必然带来内存泄露风险；\n",
    "既然内存泄露无法完全避免，当 Python 程序发生内存泄漏时，又该如何排查呢？\n",
    "\n",
    "本节，我们将以一个简单的案例，详细讲解预防、排查、解决 Python 内存泄露问题的 方法论 。\n",
    "\n",
    "工欲善其事，必先利其器。在这个过程中，我们将利用一些趁手的工具（例如 objgraph 等）。只有选择正确工具，掌握工具正确使用姿势，才能做到事半功倍。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "planned-handy",
   "metadata": {},
   "source": [
    "### 问题服务"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abandoned-theme",
   "metadata": {},
   "outputs": [],
   "source": [
    "import uvicorn\n",
    "\n",
    "from fastapi import FastAPI\n",
    "from faker import Faker\n",
    "\n",
    "from pyconsole import start_console_server\n",
    "\n",
    "faker = Faker()\n",
    "cache = {}\n",
    "\n",
    "app = FastAPI()\n",
    "\n",
    "async def fetch_user_from_database(user_id):\n",
    "    return {\n",
    "        'user_id': faker.sha256() if user_id == 'random' else user_id,\n",
    "        'name': faker.name(),\n",
    "        'email': faker.email(),\n",
    "        'address': faker.address(),\n",
    "        'desc': faker.text(),\n",
    "    }\n",
    "\n",
    "async def get_user(user_id):\n",
    "    data = cache.get(user_id)\n",
    "    if data is not None:\n",
    "        return data\n",
    "\n",
    "    data = await fetch_user_from_database(user_id)\n",
    "    cache[data['user_id']] = data\n",
    "\n",
    "    return data\n",
    "\n",
    "@app.get('/users/{user_id}')\n",
    "async def retrieve_user(user_id):\n",
    "    return await get_user(user_id)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    start_console_server()\n",
    "    uvicorn.run(app)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unlikely-smile",
   "metadata": {},
   "source": [
    "这是一个基于  框架编写的 API 服务，它只实现了一个接口：根据用户 ID 获取用户信息。API 服务由 uvicorn 启动，它是一个性能非常优秀的 ASGI 服务器。\n",
    "\n",
    "为减少数据库访问频率，程序将数据库返回的用户数据，以用户 ID 为索引，缓存在内存中 ( cache 字典)。注意到，演示服务直接使用  随机生成用户数据，模拟数据库查询，以此消除数据库依赖。\n",
    "\n",
    "顺便提一下，faker 是一个生成假数据的模块，非常好用。特别是需要测试数据时，完全不用自己绞尽脑汁拼造。\n",
    "\n",
    "服务还启动了一个远程交互式终端，以便我们可以连上服务进程，并在里面执行一些代码。交互式终端的源码可以在 github 上获得：  ，原理超过本节讨论范围不展开介绍。\n",
    "\n",
    "由于例子代码非常简单，哪里内存泄露我们甚至仅凭肉眼便可看出。尽管如此，我们假装什么都不知道，来研究解决问题的思路：如何观察程序？如何运用工具来获取一些关键信息？如何分析各个线索？如何逐步接近问题的根源？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acoustic-queens",
   "metadata": {},
   "source": [
    "### 运行服务"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blank-zambia",
   "metadata": {},
   "source": [
    "由于服务依赖几个第三方包，启动它之前请先用 pip 安装这些依赖包，并且确保安装是成功的：\n",
    "\n",
    "直接执行 service.py 即可启动服务，默认它会监听 8000 端口：\n",
    "\n",
    "    $ python service.py\n",
    "    INFO:     Started server process [76591]\n",
    "    INFO:     Waiting for application startup.\n",
    "    INFO:     Application startup complete.\n",
    "    INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n",
    "    \n",
    "服务启动后，即可通过 8000 端口访问用户信息接口，用户 ID 可以随便给：\n",
    "\n",
    "    $ curl http://127.0.0.1:8000/users/bef76936c7d22e98f3d7b4c7e1aeef524da4ec1b48f871926fee43c5ec071a2d\n",
    "    {\"user_id\":\"bef76936c7d22e98f3d7b4c7e1aeef524da4ec1b48f871926fee43c5ec071a2d\",\"name\":\"Patricia Johnson\",\"email\":\"epatton@yahoo.com\",\"address\":\"837 Jacobs Field\\nGregorybury, ND 81050\",\"desc\":\"Third choice air together expect account war. Seven dog safe significant. Expect exist wrong finish window there raise. Third blue and cover.\"}\n",
    "    \n",
    "服务接口还支持随机查询，随机返回一个用户的信息：\n",
    "\n",
    "    $ curl http://127.0.0.1:8000/users/random\n",
    "    {\"user_id\":\"d6a55f04bab8ddec83d651bdca77f7215042b792970482213b6da56a119f18a8\",\"name\":\"Evan Carter\",\"email\":\"andrea79@garcia.com\",\"address\":\"109 Miller Lights Apt. 843\\nPort Jamie, IN 97570\",\"desc\":\"Resource green allow him. Build store enough effect alone. Everybody right remember public coach book not.\\nConference respond trip girl.\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "novel-separation",
   "metadata": {},
   "source": [
    "### 远程终端"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "respiratory-editor",
   "metadata": {},
   "source": [
    "我们直接执行 pyconsole.py ，以默认端口即可连接正在运行中的 API 服务进程：\n",
    "\n",
    "    $ python pyconsole.py\n",
    "    Python 3.8.5 (default, Aug  5 2020, 18:49:57)\n",
    "    [GCC 5.4.0 20160609] on linux\n",
    "    Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n",
    "    (ConsoleClient)\n",
    "    >>>\n",
    "    \n",
    "pyconsole 用法跟 Python 交互式终端一样，但代码执行环境是在被连接的服务进程里面，因此可以看到服务内部的实时状态。我们先通过 dir 内建函数看看远程终端的名字空间都有些啥：\n",
    "\n",
    "    >>> dir()\n",
    "    ['__builtins__', '__doc__', '__name__', 'main', 'sys']\n",
    "    >>> main\n",
    "    <module '__main__' from 'service.py'>\n",
    "    >>> dir(main)\n",
    "    ['Faker', 'FastAPI', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'app', 'cache', 'faker', 'fetch_user_from_database', 'get_user', 'retrieve_user', 'start_console_server', 'uvicorn']\n",
    "    \n",
    "main 就是服务的 __main__ 模块，从中还可以找到 service.py 导入的 Faker 、FastAPI 等，它定义的函数 retrieve_user 、get_user 等，还有作为全局变量存在的 cache 字典。甚至，我们还可以看到 cache 当前缓存了多少用户信息：\n",
    "\n",
    "    >>> len(main.cache)\n",
    "    2\n",
    "    \n",
    "由于我们前面通过 API 获取了 2 条用户数据，因此 cache 当前缓存了 2 条数据。当我们再次访问接口获取其他用户数据时，我们会看到 cache 缓存的用户数据会慢慢增加：\n",
    "\n",
    "    >>> len(main.cache)\n",
    "    3\n",
    "    \n",
    "pyconsole 是一个很神奇的终端，能够实时查看 Python 进程里面各种数据的状态，在排查问题时非常方便！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bibliographic-roots",
   "metadata": {},
   "source": [
    "### 内存测量"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "outstanding-concentration",
   "metadata": {},
   "source": [
    "解决内存泄露问题，首先需要确认问题的存在性以及严重程度。程序它占用了多少内存？内存增长速度快吗？\n",
    "\n",
    "在 Linux 系统，可以执行 ps 命令查看进程信息，从中可以看到内存占用量：\n",
    "\n",
    "    $ ps aux | grep python | grep service.py\n",
    "    fasion    6467  0.2  0.5 306600 44152 pts/2    Sl+  17:02   0:03 python service.py\n",
    "    \n",
    "我们重点关注 VSZ 和 RSS ，这两者是评价进程内存的重要指标，必要时请复习操作系统相关知识力求准确理解。\n",
    "\n",
    "- VSZ ，程序虚拟内存空间大小；\n",
    "- RSS ，程序占用的物理内存大小；\n",
    "\n",
    "ps 命令输出告诉我们，服务进程刚启动时，虚拟内存空间大小大概是 306M ，占用物理内存大概 44M 。看上去没问题，但进程内存会不会随着时间推移不断增长呢？\n",
    "\n",
    "接下来，我们用  命令向服务压大量请求，看看它有什么反应。先压 1 万个请求试试看：\n",
    "\n",
    "    ab -n 10000 -c 100 http://127.0.0.1:8000/users/random\n",
    "    \n",
    "我们发现程序进程虚拟内存空间增长到 327M ，而物理内存则增长到 63M ，继续压请求发现内存还继续涨：\n",
    "\n",
    "|请求数|\t虚拟内存 / M|\t物理内存 / M|\n",
    "|:-:|:-:|:-:|\n",
    "|10000|\t327|\t63|\n",
    "|20000|\t339|\t75|\n",
    "|50000|\t365|\t101|\n",
    "|100000|\t400|\t135|\n",
    "\n",
    "随着时间的推移，新请求不断达到，进程内存也在缓缓上升。不管内存上升速度多慢，程序最终必然翻车。建议利用监控系统，对重要服务进程内存进行监控，这样可以第一时间发现内存泄露问题：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_42.jpg\" width=800 height=600> \n",
    "回放历史数据，从中可以看出进程内存的长期增长趋势。通过分析长期趋势，甚至能够精准确定内存问题出现的时间点；结合当时变更情况，准确定位问题代码，事半功倍：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_43.jpg\" width=800 height=600> \n",
    "如上图，程序虚拟内存大小在某个时间点突然陡增。联系当时相关代码变更，定位问题更有针对性。\n",
    "\n",
    "**objgraph**\n",
    "objgraph 是一个用于勘察 Python 内部对象的第三方模块，擅长绘制对象引用关系图以及排查内存泄露问题：\n",
    "\n",
    "- 取出某种类型的所有实例对象 ( by_type )；\n",
    "- 统计程序中数量最多的对象，并排序 ( show_most_common_types )；\n",
    "- 绘制对象引用关系图 ( show_refs )；\n",
    "- 绘制反向引用关系图 ( show_backrefs )；\n",
    "- etc\n",
    "\n",
    "现在，我们先调用 show_most_common_types 函数，统计数量最多的对象类型：\n",
    "\n",
    "    >>> objgraph.show_most_common_types()\n",
    "    dict                 16482\n",
    "    method               8189\n",
    "    function             7095\n",
    "    list                 6632\n",
    "    tuple                4797\n",
    "    deque                4670\n",
    "    Event                3112\n",
    "    weakref              2307\n",
    "    AsyncExitStack       1554\n",
    "    RequestResponseCycle 1553\n",
    "\n",
    "我们看到，程序中 dict 对象最多，总过有 16000 多个，method 对象次之。一般而言，存在内存泄露的程序，内部会有大量的某种对象。如果发现某个类的对象特别多，问题就可以缩小到与该类相关的代码逻辑。\n",
    "\n",
    "由于我们的程序没有定义类，因此这里看到的大部分对象都是内建类型，没有足够特异性。那是不是就无解了呢？\n",
    "\n",
    "肯定不是的。由于泄露的大量对象一定是保存在某种容器中，因此程序中应该有某个容器对象长度特别长。只要找到这个容器对象，问题也就解决了。那么，如何找到这个关键对象呢？\n",
    "\n",
    "常用的容器对象无非 tuple 、list 、dict 这几个，我们只需将这几个种类的对象找到即可。objgraph 有一个函数 by_type ，用于获取某个种类的对象，刚好可以派上用场。我们先将程序中所有 tuple 对象找出来：\n",
    "\n",
    "    >>> tuples = objgraph.by_type('tuple')\n",
    "    >>> len(tuples)\n",
    "    4425\n",
    " 如此一来，我们可以写一个函数将长度最长的给定类型对象找出来：\n",
    "\n",
    "      def get_max_instance(type_name):\n",
    "        # 取出所有实例\n",
    "        instances = objgraph.by_type(type_name)\n",
    "        if not instances:\n",
    "            return\n",
    "\n",
    "        # 临时记录最大大小\n",
    "        maxsize = 0\n",
    "        max_one = None\n",
    "\n",
    "        # 遍历实例\n",
    "        for instance in instances:\n",
    "            size = len(instance)\n",
    "            if size > maxsize:\n",
    "                maxsize = size\n",
    "                max_one = instance\n",
    "\n",
    "        return max_one \n",
    "\n",
    "接着，我们将每种容器中，长度最大的对象找出来：\n",
    "\n",
    "    >>> max_tuple = get_max_instance('tuple')\n",
    "    >>> len(max_tuple)\n",
    "    42\n",
    "    >>> max_list = get_max_instance('list')\n",
    "    >>> len(max_list)\n",
    "    65535\n",
    "    >>> max_dict = get_max_instance('dict')\n",
    "    >>> len(max_dict)\n",
    "    85865\n",
    "    \n",
    "最大的 tuple 长度只有 42 ，肯定与问题无关；剩下的 list 和 dict ，dict 更大一点，严重怀疑它。先来看看它里面都保存着什么东西：\n",
    "\n",
    "    >>> for k, v in max_dict.items():\n",
    "    ...     break\n",
    "    ...\n",
    "    >>> k\n",
    "    '956712794f14a1377352f77c844c89fe75c96fb1bf7cd3220e11e31ff8761d67'\n",
    "    >>> v\n",
    "    {'user_id': '956712794f14a1377352f77c844c89fe75c96fb1bf7cd3220e11e31ff8761d67', 'name': 'Luis Lewis', 'email': 'tammyhale@yahoo.com', 'address': '24043 Javier Summit Apt. 283\\nEdwardchester, CA 19580', 'desc': 'Trip alone material because doctor. Story play night between entire interview as dog.\\nMeasure actually law high. Discussion at project would argue knowledge land direction.'}\n",
    "    \n",
    "由于我们的服务程序非常简单，以 ID 为 key ，以用户信息为 value 的字典就只有 cache 了。由此，我们发现了问题的所在，正是 cache 字典长度不断增长，导致服务占用内存越来越多。\n",
    "\n",
    "再退一步，如果服务逻辑非常复杂，无法肉眼看出这个巨大的 dict 是什么东西，又该如何定位问题呢？莫慌，objgraph 可以帮忙找出引用这个 dict 的地方：  \n",
    "\n",
    "    >>> chain = objgraph.find_backref_chain(max_dict, objgraph.is_proper_module)\n",
    "    \n",
    "find_backref_chain 函数将找出 max_dict 对象的反向引用链，引用链以 list 的形式返回。is_proper_module 告诉它，当回溯遇到模块对象后，停止搜索。Python 模块对象辨识度很高，从模块对象出发，沿着引用链即可发现 max_dict 的来龙去脉。\n",
    "\n",
    "在这个例子，引用链非常简单，只有区区 3 个节点：\n",
    "\n",
    "    >>> len(chain)\n",
    "    3\n",
    "    >>> chain[-1] is max_dict\n",
    "    True\n",
    "    \n",
    "其中，最后一个节点就是我们要定位的 max_dict 字典；第一个节点是 __main__ 模块。由此，我们可以断定： max_dict 是 __main__ 模块中的某个属性，而第二个节点就是 __main__ 模块的属性空间：\n",
    "<img style=\"float: middle;\" src=\".\\imgs\\C_07\\C7_44.jpg\" width=800 height=600> \n",
    "如果引用链较长，建议将它画在一个图片上进行分析，更为直观。objgraph 内置生成图片的功能，不过需要先安装 graphviz 以及 xdot 这两个依赖。注意 graphviz 与 xdot 不同，它不是 Python 包，以 Ubuntu 为例用 apt 安装：\n",
    "\n",
    "依赖安装完毕后，调用 show_chain 函数进行绘图，filename 参数指定图片名字：\n",
    "\n",
    "    >>> objgraph.show_chain(chain, filename='chain.png')\n",
    "    Graph written to /tmp/objgraph-szst76n5.dot (3 nodes)\n",
    "    Image generated as chain.png\n",
    "    \n",
    "如果模块对象非常复杂，包含很多字典属性，难以区分。我们就只能逐个遍历对比，才能确定问题属性：\n",
    "\n",
    "    >>> module, attrs = chain[:2]\n",
    "    >>> for name, value in attrs.items():\n",
    "    ...     if value is max_dict:\n",
    "    ...         print(name)\n",
    "    ...         break\n",
    "    ...\n",
    "    cache\n",
    "    \n",
    " 至此，通过种种手段，我们已经百分之百确定 —— 问题字典就是 __main__ 模块中用于缓存数据的 cache ：\n",
    "\n",
    "    >>> max_dict is main.cache\n",
    "    True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "equipped-solid",
   "metadata": {},
   "source": [
    "### 未雨绸缪"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "awful-short",
   "metadata": {},
   "source": [
    "扁鹊曰：长兄于病视神，未有形而除之，故名不出于家。所谓解决患难者强，防患于未然者神。那么，对于内存泄露隐患，是否也能够做到防范于未然呢？答案是肯定的。\n",
    "\n",
    "首先，针对造成内存泄露的根源，我们在写代码时要时刻牢记以下两点：\n",
    "\n",
    "- 往容器添加元素时，要考虑退出 (删除) 机制；\n",
    "- 慎用 __del__ 魔术方法；\n",
    "\n",
    "其次，做好程序内存监控。将程序占用内存大小，每种容器对象的最大长度等关键指标提交到监控系统，并绘制趋势图。这样一来，趋势图可以帮我们发现一些潜在的问题，将它解决在爆发前夜。此外，每次服务变更时，勿忘关注内存使用趋势，异常可在第一时间发现。\n",
    "\n",
    "最后，我们还可以事先在程序中定义一些常用工具函数。一旦程序出现内存泄露问题，我们可以通过远程终端连上服务进程实时诊断，这些工具函数将极大加速问题定位的速度。下面是一个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "convertible-timber",
   "metadata": {},
   "outputs": [],
   "source": [
    "import objgraph\n",
    "import random\n",
    "\n",
    "\n",
    "def show_instance_randomly(type_name):\n",
    "    # 类型名及图片输出路径\n",
    "    type_name = str(type_name)\n",
    "    filename = '/tmp/%s.png' % (type_name,)\n",
    "    \n",
    "    # 取出所有实例\n",
    "    instances = objgraph.by_type(type_name)\n",
    "    if not instances:\n",
    "        return\n",
    "    \n",
    "    # 随机取出一个实例\n",
    "    instance = random.choice(instances)\n",
    "    \n",
    "    # 画反向引用图\n",
    "    objgraph.show_backrefs([instance], filename=filename)\n",
    "\n",
    "\n",
    "def get_max_instance(type_name):\n",
    "    # 取出所有实例\n",
    "    instances = objgraph.by_type(type_name)\n",
    "    if not instances:\n",
    "        return\n",
    "    \n",
    "    # 临时记录最大大小\n",
    "    maxsize = 0\n",
    "    max_one = None\n",
    "    \n",
    "    # 遍历实例\n",
    "    for instance in instances:\n",
    "        size = len(instance)\n",
    "        if size > maxsize:\n",
    "            maxsize = size\n",
    "            max_one = instance\n",
    "    \n",
    "    return max_one\n",
    "\n",
    "\n",
    "def show_max_instance(type_name):\n",
    "    # 类型名及图片输出路径\n",
    "    type_name = str(type_name)\n",
    "    filename = '/tmp/max_%s.png' % (type_name,)\n",
    "    \n",
    "    # 找出最大实例\n",
    "    max_one = get_max_instance(type_name=type_name)\n",
    "    \n",
    "    # 反向引用图\n",
    "    objgraph.show_backrefs([max_one], filename=filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spectacular-local",
   "metadata": {},
   "source": [
    "- show_instance_randomly ，随机取出一个指定类型对象，生成反向引用图，保存在 /tmp 目录；\n",
    "- get_max_instance ，搜索长度最大的指定类型容器对象；\n",
    "- show_max_instance ，搜索长度最大的指定类型容器对象，生成反向引用图，保存在 /tmp 目录；"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "1116px",
    "left": "0px",
    "top": "92px",
    "width": "376.594px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}